KB semantics phase 2 — section clustering + explorer — design

Cluster manual SECTIONS that mean the same thing across manuals (semantic dedup / overlap discovery), and expose the clusters as a browsable web page. Fully local. Foundation for phase 3 (claude -p consolidated generation).

Roadmap position

Phase 1 (DONE): vectorize — 3,252 chunk vectors across ~12 manuals + kick-fix in Qdrant manuals.
Phase 2 (THIS): section-level semantic clustering + web explorer. 100% local, no LLM.
Phase 3 (LATER): claude -p consolidated generation. clusters.json from phase 2 is its retrieval bundle (each cross-manual cluster = “every source covering topic X”). Needs Claude installed+logged-in on the box (not done).

Goal

Group manual sections that describe the same procedure/topic across different manuals, so you can see e.g. “differential replacement is covered in the workshop §7B, geo-tracker, kick-fix/axles, and sidekick.” Expose as a web page where each cluster lists its source sections as links into the knowledgebase.

Non-goals

No LLM / claude -p in phase 2 (labels are local; generation is phase 3).
No chunk-level dedup (cluster whole sections, not passages).
No re-embedding (reuse phase-1 vectors).
No modification of the manuals or the vector index (read-only over Qdrant).
Not an always-on new service (fold the explorer into the existing knowledgebase app).

Key decisions

Decision	Choice
Cluster unit	manual SECTION (one `NNN-*.md` page)
Section vector	mean-pool the section’s chunk vectors, re-normalize
Algorithm	HDBSCAN (auto cluster count, noise bucket), one knob `min_cluster_size`
Label	representative section’s heading (nearest cluster centroid) + source-manual list — LOCAL, pluggable
Ranking	by cross-manual span (distinct manuals a cluster touches), dedup targets first
Artifact	`clusters.json`
Explorer	`/semantics` route in the existing knowledgebase Flask app (no new service)
Dashboard	add a “Szemantika” tile → `/semantics`
Runs as	`levander`, thread-capped, on demand

Architecture

Extends /home/levander/kb-vectors/ (reuses its venv, Qdrant client, and the manuals collection). Adds sklearn + hdbscan to the venv.

~/kb-vectors/
  sectionvecs.py   pure-ish: pull all points from Qdrant -> group by (manual_id, page)
                   -> mean-pool chunk vectors -> normalized section vectors + section metadata
  cluster.py       HDBSCAN over section vectors -> clusters; label + cross-manual span;
                   write clusters.json
  labels.py        pure: representative-section pick + label string (pluggable; claude relabel later)
  kbclust.py       CLI: `build` (regenerate clusters.json), `list`, `show <id>`
  clusters.json    the artifact (consumed by the explorer)

~/knowledgebase/
  app.py           + GET /semantics (renders clusters.json), + GET /api/clusters (json)
  (dashboard)      home-portal: add a "Szemantika" tile linking to knowledgebase /semantics

Data flow

Qdrant `manuals` (chunk vectors + payload {manual_id, page, page_url, heading, images})
  -> group points by (manual_id, page)              [a "section"]
  -> mean-pool the chunk vectors of each section, L2-normalize
  -> section_vectors[N], section_meta[N]            [N ~= 400-600 sections]
  -> HDBSCAN(metric=euclidean on normalized vectors ~ cosine)   [min_cluster_size=2]
  -> for each cluster: members, representative (nearest centroid), label, source_manuals, span
  -> clusters.json: [{id, label, size, span, source_manuals, members:[{manual_id, page_url, heading}]}]
  -> /semantics reads clusters.json -> HTML list, members link to <kb>/<page_url>

Components

sectionvecs.py

load_sections(client) -> (vectors: np.ndarray[N,1024], meta: list[dict]) — scroll all points from manuals (with vectors + payload), group by (manual_id, page), mean-pool each group’s vectors, L2-normalize. meta[i] = {manual_id, folder, page, page_url, heading} (heading/page_url taken from any chunk of that section; they’re identical per section).
Deterministic ordering (sort sections by manual_id, then page) so runs are reproducible.

cluster.py

build_clusters(vectors, meta, min_cluster_size=2) -> list[dict]:
- hdbscan.HDBSCAN(min_cluster_size=min_cluster_size, metric="euclidean") on the normalized vectors (unit-norm + euclidean ≈ cosine ordering).
- For each label != -1: gather member indices; centroid = normalized mean of members; representative = member nearest the centroid; source_manuals = sorted distinct manual_ids; span = len(source_manuals).
- Label via labels.make_label(rep_meta, source_manuals).
- Noise (label == -1) is excluded from clusters (optionally reported as a count).
- Return clusters sorted by span desc, then size desc. Assign stable id by that order.
write_clusters(clusters, path) / load_clusters(path).

labels.py (pure, pluggable)

representative_index(member_indices, vectors) -> int — index nearest the member centroid.
make_label(rep_meta, source_manuals) -> str — the representative section’s heading (fallback: page slug prettified if heading empty), trimmed. The explorer shows this + the source-manual list separately, so the label need not encode sources.
This is the single seam where a future claude -p relabel drops in behind the same signature.

kbclust.py (CLI)

build [--min-cluster-size N] — load sections, cluster, write clusters.json; print counts (sections, clusters, cross-manual clusters, noise).
list — print clusters (id, span, size, label).
show <id> — print a cluster’s members (manual_id, heading, page_url).

knowledgebase app additions

GET /api/clusters — serve clusters.json (from /home/levander/kb-vectors/clusters.json; if missing, {clusters: []}).
GET /semantics — render an HTML page: title, a summary line (N clusters, M cross-manual), then each cluster: label, a badge of span sources, and its members as links to /<page_url> (same-origin KB section pages). Cross-manual clusters first. Reuses the app’s existing style; path-safe (read-only file read, no user input into paths).
Dashboard home-portal/index.html: add a tile “Szemantika” → https://knowledgebase.taild4189d.ts.net/semantics.

Failure / ops notes

Read-only over Qdrant and the KB docs; cannot harm either.
Thread-capped (OMP_NUM_THREADS=4) + nice; clustering a few hundred vectors is quick and CPU-light (per the PSU-load lesson, still serialize vs other heavy jobs).
Re-runnable: kbclust build regenerates clusters.json after the corpus changes; the explorer just re-reads the file.
If clusters.json is missing/stale, /semantics shows an empty state with a “run build” hint rather than erroring.
No new always-on service; the explorer is a route on the already-running knowledgebase app.

Testing / verification

sectionvecs unit: a fake set of points across 2 sections → 2 normalized mean-pooled vectors with correct meta; unit-norm verified.
labels unit: representative pick returns the nearest-centroid member; make_label returns the heading (and the empty-heading fallback).
cluster unit: synthetic vectors in 2 obvious groups → 2 clusters, correct members, span/source_manuals computed, sorted by span.
Real build: run kbclust build on the live corpus; assert non-trivial cluster count and that some clusters are cross-manual (span ≥ 2).
Quality proof (the real one): known-overlapping topics land together across manuals — find the cluster containing a workshop differential section and assert it also contains geo-tracker and/or kick-fix/axles; likewise a brake-bleeding and an O2-sensor cluster spanning manuals.
Explorer: /semantics returns 200, lists clusters cross-manual-first, and a member link resolves to a real KB section page (200).
Dashboard: the Szemantika tile is present and points at /semantics.

Risks

Risk	Mitigation
HDBSCAN too coarse/fine	tune `min_cluster_size` (start 2); it’s a single documented knob
Terse OCR headings make weak labels	show heading + source list; pluggable label seam for a later `claude` relabel
Sparse/diagram sections (image-only docs) cluster weakly	HDBSCAN routes them to the noise bucket; report noise count, don’t force them
Section mean-pool loses within-section nuance	acceptable at section granularity (chosen unit); chunk-level is a later option
Cross-boot instability during build	short job, thread-capped, re-runnable; artifact on disk survives

Levandor

Explorer

KB semantics phase 2 — section clustering + web explorer — design