Levandor

KB semantics (phase 2) — section clustering + web explorer — Built

6 min read

As-built completion record for phase 2 of the knowledgebase semantics work: fully-local semantic SECTION clustering over the phase-1 vectors plus a read-only web explorer (“Szemantika”) folded into the existing knowledgebase Flask app. No LLM. Result: 194 clusters (post boilerplate filter), 24 cross-SOURCE (span ≥ 2), 631 noise singletons from ~1248 section vectors mean-pooled out of 3252 chunk vectors. This is the retrieval-bundle foundation for phase 3 (claude -p consolidated generation), which is still NOT started.

Design in 2026-07-24-kb-semantics-phase2-design, task-by-task plan in 2026-07-24-kb-semantics-phase2-plan — this note is the completion record and the hard-won knowledge, not a restatement of either. Builds on 2026-07-24-kb-vectorize-complete (phase 1) and reads the same 2026-07-24-knowledgebase library.

For Agents

Code: /home/levander/kb-vectors/ on telep-mainframe (reuses the phase-1 venv, runs as levander). New modules: sectionvecs.py, labels.py, cluster.py, kbclust.py (+ tests). CLI: kbclust.py build / list / show <id>. Pipeline: scroll all 3252 chunk vectors from Qdrant collection manuals (READ-ONLY) → group by (manual_id, page) → L2-normalized mean-pool → ~1248 SECTION vectors → HDBSCAN(min_cluster_size=2, min_samples=1, metric="euclidean") on unit-norm vectors (≈ cosine) → representative-heading labels → clusters.json. Artifact: /home/levander/kb-vectors/clusters.json, regenerated on demand via kbclust.py build (thread-capped). Explorer: GET /semantics + GET /api/clusters added to the existing knowledgebase Flask app (/home/levander/knowledgebase/app.py) — NO new always-on service. Live at https://knowledgebase.taild4189d.ts.net/semantics (Hungarian UI “Szemantika”). Dashboard tile 🧭 Szemantika on home.taild4189d.ts.net. Ranking: clusters sorted by cross-SOURCE span (dedup targets surface first).

What was built

  • Section-level clustering pipeline (sectionvecs.py + cluster.py + labels.py): pulls the phase-1 chunk vectors out of Qdrant, mean-pools each (manual_id, page) group into one L2-normalized SECTION vector (3252 chunks → ~1248 sections), then HDBSCAN-clusters them into topics that recur across manuals.
  • kbclust.py CLI: build (regenerate clusters.json, prints section/cluster/cross-source/noise counts), list (id / span / size / label), show <id> (members with manual_id + heading + page_url).
  • Web explorer /semantics + JSON /api/clusters on the existing knowledgebase app — each cluster shows its label + N források badge + member sections linking straight into the KB pages. Cross-source clusters first. Read-only file read of clusters.json, no user input into paths.
  • Dashboard tile 🧭 Szemantika on the home.taild4189d.ts.net portal → the explorer.

Result (the real proof)

  • 194 clusters after the boilerplate filter (22 dropped), 24 cross-SOURCE (span ≥ 2 distinct top-level source folders), 631 noise singletons.
  • Top clusters are genuine cross-make overlaps: WINDOW REGULATOR spanning chevy-tracker + suzuki-vitara/5door + workshop; ENGINE COOLING; PARKING BRAKE; THROTTLE BODY. These are exactly the “same topic covered in N manuals” dedup targets phase 3 will consolidate.

Why these choices

  • SECTION as the cluster unit, not chunk — cluster whole procedures/topics, not passages; one NNN-*.md KB page = one section, mean-pooled from its chunk vectors.
  • Local heading labels, pluggable — the label is the representative (nearest-centroid) section’s heading, behind a single make_label seam. A future claude -p relabel pass drops in there for nicer titles without touching the clustering. See the tip below.
  • Explorer folded into the existing knowledgebase app — deliberately NO new always-on service on the power-flaky box (see 2026-07-24-psu-load-resets and the mains die-off). It’s just a route on an already-running app; if clusters.json is missing it shows an empty state, not an error.
  • Ranked by cross-SOURCE span — dedup targets (topics covered by the most distinct sources) surface first, which is the whole point of the phase.

Hard-won gotchas

HDBSCAN's min_samples defaults to min_cluster_size — leaving it default made ALL data noise (0 clusters)

With min_cluster_size=2 and min_samples left unset, HDBSCAN uses min_samples = min_cluster_size = 2, which routed every section to the noise bucket → 0 clusters. You must set min_samples=1 explicitly for min_cluster_size=2 to actually form clusters. This looked like “the data has no structure” but was purely the default-coupling of the two knobs.

"Cross-manual" span was inflated because manual_id is <folder>/<manual> and the kick-fix archive is ONE source split into ~25 topic-subfolders

Ranking by distinct manual_id made the kick-fix archive look like ~25 “different manuals” all covering the same topic — inflating span. Fix: define span by distinct TOP-LEVEL source folder (source_of = manual_id.split("/", 1)[0]), so kick-fix collapses to 1 source while genuinely separate manuals (suzuki-vitara/workshop vs suzuki-vitara/5door-supplement) still count distinctly. Keep the full source_manuals list for display, but rank by (span = distinct sources, #manuals, size).

  • Boilerplate pollution of the top ranks. Span-ranking first surfaced cover/title pages (“SUZUKI”) and generic headings (“GENERAL DESCRIPTION”, “DIAGNOSIS”) — they appear in every manual so they span the most sources. Added an is_boilerplate() label stoplist filter → 22 clusters dropped, all genuinely generic. The heading-based labels are only as good as the OCR headings — which is exactly why the label layer is pluggable for a future claude relabel.
  • 51% noise (631/1248) is EXPECTED and correct for a dedup goal. Singleton sections unique to one manual aren’t duplicated knowledge, so HDBSCAN routing them to noise is the right answer, not a bug. Don’t chase “reduce the noise” — the noise bucket is the not-duplicated set.

Read-only scroll discipline + the pluggable-label seam

The whole pipeline only scrolls points OUT of Qdrant and writes a local clusters.json — it never upserts / deletes / creates. Verified in review. This means clustering can be re-run freely against the live manuals collection with zero risk to the phase-1 index. Separately, make_label(rep_meta, source_manuals) is the one seam where a future claude -p relabel replaces heading-based titles — same signature, no change to clustering or the explorer.

Ops

  • Regenerate on demand: kbclust.py build (thread-capped per the PSU lesson — 2026-07-24-psu-load-resets) after the corpus changes; the explorer just re-reads clusters.json.
  • Read-only over Qdrant + KB docs — cannot harm either.
  • No new always-on service; the explorer is a route on the already-running knowledgebase app.

Roadmap position

  • Phase 1 (vectorize) — DONE (2026-07-24-kb-vectorize-complete): 3252 chunk vectors in Qdrant manuals.
  • Phase 2 (this — cluster + explore) — DONE.
  • Phase 3 (claude -p consolidated generation) — NOT started. clusters.json is its retrieval bundle: each cross-source cluster = the exact set of sources to hand claude for a topic. Phase 3 needs Claude Code installed + logged-in (subscription) on the box — still not done.
  • Optional bridge: a claude -p relabel pass behind make_label for nicer cluster titles.

Graph View

Backlinks