Levandor

2026-07-29-kb-semantic-search-frontend-plan

7 min read

KB Semantic-Search Frontend — Implementation Plan

For agentic workers: execute task-by-task via subagent-driven development. Each task ends with a runnable stdlib unittest check or a stated live verification. NO git commits, NO code comments (house rules). Steps use - [ ].

Goal: Add server-side semantic search (existing Qdrant + bge-large) as the KB’s hero, served by the existing Flask app, replacing the slow client-side lunr search as the primary entry.

Architecture: Extend /home/levander/knowledgebase/app.py. Reuse kb-vectors/search.py + embed.py (bge-large singleton) + Qdrant manuals. New /api/search JSON route + a search-first home. Keep mkdocs static pages served by the existing catch-all.

Design spec: 2026-07-29-kb-semantic-search-frontend-design. Related: 2026-07-24-kb-vectorize-complete, telep-mainframe.

Global Constraints

  • Work on telep-mainframe as levander; /usr/bin/ssh only. NO git commit. NO code comments. One decl/line; no inline compound statements.
  • App dir /home/levander/knowledgebase/ (venv there; Flask+waitress @127.0.0.1:8092, Restart=always systemd knowledgebase.service). Restart: sudo systemctl restart knowledgebase.
  • Semantic backend /home/levander/kb-vectors/: search.search(client, query, limit=8, folder=None, manual=None) -> [payload+score]; embed.embed_query(text) (lazy singleton _get_model(), bge-large CPU); index.COLLECTION="manuals"; Qdrant QdrantClient(host="127.0.0.1", port=6333). Payload fields: folder, manual, manual_id, page, page_url, heading, images, text, chunk_index, score.
  • Tests: stdlib unittest, cd /home/levander/knowledgebase && venv/bin/python -m unittest <module>.
  • NON-DESTRUCTIVE: don’t modify Qdrant, docs, the mkdocs site/, or clusters.json. Back up app.py before editing (.bak-search-fe). No new always-on service — everything runs inside the existing Flask process.
  • PSU limit: cap embedding threads (torch.set_num_threads(4) / OMP_NUM_THREADS=4) so concurrent encodes don’t brown-out the box.
  • bge-large is English-only (manuals are English). HU UI chrome via existing i18n.py.

Task 0: Search deps + import wiring (GO/NO-GO foundation)

Purpose: make the knowledgebase venv able to import kb-vectors/search.py (which imports embed → sentence_transformers/torch, and qdrant_client). If these can’t install cleanly, pivot to a tiny localhost search service (documented fallback) — decide here.

Files: none yet (env + a probe script /home/levander/knowledgebase/_probe_search.py, delete after).

Steps:

  • Check what the knowledgebase venv already has: venv/bin/python -c "import qdrant_client, sentence_transformers, torch; print('have all')" — if it errors, note which are missing.
  • Install the missing ones (CPU torch, to match kb-vectors): venv/bin/pip install qdrant-client sentence-transformers (torch CPU comes as a dep; if it tries CUDA, pin the CPU wheel like kb-vectors uses — check kb-vectors/venv/bin/pip freeze | grep -iE "torch|sentence|qdrant" and match those versions to avoid a heavy/incompatible pull). Prefer matching kb-vectors’ pinned versions.
  • Probe import from the app venv with kb-vectors on the path:
import sys
sys.path.insert(0, "/home/levander/kb-vectors")
import search, embed
from qdrant_client import QdrantClient
c = QdrantClient(host="127.0.0.1", port=6333)
hits = search.search(c, "fuel cut controller", limit=3)
print(len(hits), hits[0]["page_url"] if hits else "none")

Run: venv/bin/python _probe_search.py → expect ≥1 hit with a page_url.

  • GATE: if the probe returns hits → proceed (deps + import path work in-process). If install is infeasible/breaks the venv → STOP, restore, and switch the plan to the fallback (a small search_service.py run under kb-vectors venv on 127.0.0.1:8car, proxied by the app) — report which path.
  • Record the working versions + the exact sys.path line needed. Delete _probe_search.py.

Task 1: searchui.py pure helpers

Files: Create /home/levander/knowledgebase/searchui.py, test_searchui.py.

Interfaces produced:

  • dedupe_by_page(hits) -> hits — keep best-score hit per page_url, preserve order by score desc.
  • make_snippet(text, query, width=220) -> str — window around the first query-term hit (fallback: head of text), collapsed whitespace.
  • highlight(snippet, query) -> str — wrap query terms in <mark>…</mark> (case-insensitive, HTML-escape the rest).
  • classify(manual_id, page_url) -> "article"|"wiring"|"manual" — consolidated/ → article; wiring manuals (wiring-diagrams, 5door-supplement, or a page with a sibling .wires.txt) → wiring; else manual.
  • vehicle_label(folder) -> str — prettify top folder (suzuki-vitara→“Suzuki Vitara”, chevy-tracker→“Chevy/Geo Tracker”, kick-fix→“kick-fix”, …).

Steps:

  • Write test_searchui.py: dedupe keeps the higher score per page + drops dupes; snippet centers on the query term and respects width; highlight wraps terms + escapes HTML (no XSS from text); classify maps the three cases; vehicle_label maps known folders + falls back.
  • Run venv/bin/python -m unittest test_searchui → FAIL (no module).
  • Implement searchui.py; rerun → PASS.

Task 2: /api/search route + startup singletons

Files: Modify /home/levander/knowledgebase/app.py (back up .bak-search-fe). Test: test_api_search.py.

Interfaces produced: GET /api/search?q=&folder=&manual=&limit= → {took_ms, count, hits:[{heading, manual_id, vehicle, page_url, snippet, score, type}]}.

Steps:

  • In app.py: import sys; sys.path.insert(0, "/home/levander/kb-vectors") (from Task 0), import search as kbsearch, embed; import searchui; add module globals QCLIENT = QdrantClient(host="127.0.0.1", port=6333) and a warmed embedder: in main() before serve(...), call embed.embed_query("warmup") (loads the singleton) and torch.set_num_threads(4) (guard import).
  • Add route:
@app.route("/api/search")
def api_search():
    q = (request.args.get("q") or "").strip()
    if not q:
        return jsonify({"took_ms": 0, "count": 0, "hits": []})
    folder = request.args.get("folder") or None
    manual = request.args.get("manual") or None
    limit = min(int(request.args.get("limit", 20)), 50)
    import time
    t0 = time.monotonic()
    try:
        raw = kbsearch.search(QCLIENT, q, limit=limit, folder=folder, manual=manual)
    except Exception as error:
        return jsonify({"error": str(error)[:200]}), 503
    hits = searchui.dedupe_by_page(raw)
    out = []
    for h in hits:
        out.append({
            "heading": h.get("heading") or h.get("page"),
            "manual_id": h.get("manual_id"),
            "vehicle": searchui.vehicle_label(h.get("folder", "")),
            "page_url": "/" + h.get("page_url", "").lstrip("/"),
            "snippet": searchui.highlight(searchui.make_snippet(h.get("text", ""), q), q),
            "score": round(h.get("score", 0.0), 3),
            "type": searchui.classify(h.get("manual_id", ""), h.get("page_url", "")),
        })
    took = int((time.monotonic() - t0) * 1000)
    return jsonify({"took_ms": took, "count": len(out), "hits": out})
  • test_api_search.py: monkeypatch kbsearch.search to return fake payloads → assert JSON shape, dedupe applied, empty q → empty hits, filters passed through, a raised error → 503. (Use app.test_client(); stub so no real Qdrant/model needed.)
  • Run venv/bin/python -m unittest test_api_search → PASS.

Task 3: Search-first home UI + / override

Files: Modify app.py (add SEARCH_HTML, override /, add /browse). Update test_app.py if it asserts on /.

Steps:

  • Add SEARCH_HTML (inline render_template_string, dark theme matching SEMANTICS_HTML): autofocused search box; JS that debounces (~200 ms) input → fetch("/api/search?q=...&folder=...") → renders results (type badge, heading link to page_url — for type=="wiring" append #kinyert-huzalozasi-adatok-ai, vehicle/manual, highlighted snippet, score); vehicle filter <select>; result count + took_ms; ↑/↓/Enter keyboard nav; i18n chrome via i18n.strings(current_lang()). Works with JS off via a GET <form action="/search"> fallback (server renders results by calling the same logic).
  • Change routing: remove @app.route("/") from the site() catch-all (leave @app.route("/<path:path>")), add @app.route("/") → render SEARCH_HTML; add @app.route("/search") → render SEARCH_HTML (optionally server-render results if q present, for no-JS). Add @app.route("/browse") → send_from_directory(SITE, "index.html") so the mkdocs manual index stays reachable, and link it from the search home (“Böngészés / Browse manuals”).
  • Add i18n keys for the new chrome (search_placeholder, results_count, no_results, browse_manuals, filter_vehicle) to i18n.py (hu + en).
  • Update test_app.py: / now returns the search UI (contains the search box marker) not the mkdocs index; /browse serves the mkdocs index; a manual page path still 200s via the catch-all.
  • venv/bin/python -m unittest (full suite) → PASS.

Task 4: Live integration + verify + ship

Steps:

  • sudo systemctl restart knowledgebase; confirm it comes up (waitress serving; embedder warmed at startup — check journalctl -u knowledgebase -n 20 for a clean start, note startup time).
  • Live queries: curl -s "localhost:8092/api/search?q=fuel+cut+controller" | python3 -m json.tool | head → relevant hits (right pages) ranked; try “why won’t the fuel pump prime”, “stop light switch”, a folder=suzuki-vitara filtered query → restricted. Report top results + took_ms (< ~500 ms warm).
  • Wiring result deep-links to its #kinyert-huzalozasi-adatok-ai section; article result points into consolidated/.
  • Home: curl -s localhost:8092/ | grep -c "<search box marker>" = 1; /browse serves the mkdocs index (200); an existing manual page still 200; /semantics, /drafts still 200 (no regression).
  • Graceful degradation: docker stop kb-qdrant (or the Qdrant container) → /api/search returns 503, / still renders, mkdocs pages still serve; then docker start it back.
  • Memory check: ps -o rss= -p $(pgrep -f "python app.py") ~1.3–1.6 GB resident (embedder) — sane vs 42 GB free.
  • Confirm non-destructive: Qdrant/docs/site/clusters.json untouched; backups present (app.py.bak-search-fe).

Self-review

  • Spec coverage: /api/search (T2), semantic backend reuse (T0/T2), search-first home + filters + deep links + badges (T3), keep mkdocs pages (T3 catch-all + /browse), i18n chrome (T3), graceful degradation (T4), no synthesis (retrieval only, T2). Covered.
  • Ordering de-risks: T0 gates the in-process import (fallback to a service if deps won’t install) before any UI work.
  • Type consistency: searchui.dedupe_by_page/make_snippet/highlight/classify/vehicle_label defined T1, consumed T2; /api/search JSON shape defined T2, consumed by T3 UI.
  • No placeholders; every code step shows real code or exact commands.

Graph View

Backlinks