KB cluster coherence-judge + relabel (phase 2.5) — design

Add a claude -p pass after the local HDBSCAN clustering that judges each cluster’s coherence, drops incoherent grab-bags, prunes tangential members, and writes clean labels. Fixes the “painting guides inside the Window Regulator cluster” problem.

Related: 2026-07-24-kb-semantics-phase2-complete, 2026-07-25-kb-consolidated-gen-phase3-complete, telep-mainframe, SESSION-HANDOVER

Problem (root-caused)

The phase-2 explorer surfaces grab-bag clusters: e.g. cluster 0 “WINDOW REGULATOR” (size 8) also contains “Filling and painting”, “Finding an open circuit”, “Body Service”, “Rear View Mirror”. Investigation showed: bge section embeddings compress all terse technical-manual prose into a 0.87–0.98 cosine cone, so density clustering merges topically-different but stylistically-similar sections. Neither HDBSCAN params (leaf vs eom identical), nor cosine cohesion, nor lexical-heading overlap cleanly separates grab-bags from real clusters. Embeddings can’t discriminate automotive subtopics here.

Proof: claude -p opus fed the members judged cluster 0 coherent:false (drop all) and the parking-brake cluster coherent:true with a clean label and pruned tangential members. Meaning-judgement is exactly what the embeddings lacked.

Goal

A trustworthy /semantics explorer: incoherent clusters gone, remaining clusters pruned to genuinely-related sections, with clean human labels (replacing terse OCR headings like **DIAGNOSIS** / 5. PARKING BRAKE INSPECTION AND ADJUJSTMENT).

Non-goals

• No change to the base HDBSCAN clustering (stays fully local; judge is a post-pass).
• No splitting of grab-bags into sub-clusters (v1: drop incoherent; splitting is a later option).
• No re-embedding, no change to Qdrant / phase-1 / phase-3.
• No change to the explorer route/UI (it reads the same clusters.json shape).

Key decisions

Decision	Choice
Scope	Judge ALL clusters (not just cross-source) — grab-bags exist below the fold
Action	Drop incoherent (or <2 kept); refine coherent (apply keep_members); relabel all
Split grab-bags	No (v1) — drop only
Model	claude -p --model opus, batched (~15 clusters/call) to keep to ~13 calls
Base clustering	Unchanged, local; judge is an added pass
Artifact	Overwrites clusters.json (with a .raw.json backup of the pre-judge output)
Re-run	kbclust judge (separate from build); or build then judge
Runs as	levander, serialized, thread-capped (box power-flaky + claude heavy)

Architecture

Extends /home/levander/kb-vectors/ (where clustering + clusters.json live). claude is at /home/levander/.local/bin/claude, logged in as levander.

~/kb-vectors/
  judgeprompt.py   pure: build the batch judge prompt from N clusters; parse claude's JSON reply
  judge.py         run: load clusters.json -> batch -> claude -p opus -> apply verdicts
                   (drop/refine/relabel) -> recompute span/source_manuals from kept members
                   -> write clusters.json (+ clusters.raw.json backup)
  kbclust.py       + `judge [--batch N]` subcommand

Data flow

clusters.json (raw, from build)  ->  batches of ~15 clusters
  each batch -> claude -p opus (judgeprompt) -> JSON array:
     [{index, coherent, label, keep_headings:[...]}, ...]
  apply per cluster:
    - coherent==false -> DROP
    - keep only members whose heading is in keep_headings (match on heading text)
    - if kept members < 2 -> DROP
    - label = claude's clean label
    - recompute source_manuals (distinct manual_id of kept), sources (distinct top-level folder),
      span = len(sources), size = len(kept members)
  -> surviving clusters, re-sorted by (span, #manuals, size) desc, ids reassigned sequentially
  -> write clusters.json ; keep clusters.raw.json as the pre-judge snapshot

Components

judgeprompt.py (pure, testable)

• build_batch_prompt(clusters) -> str — one prompt describing the task + each cluster labeled CLUSTER <index>: with its members - [<manual_id>] <heading>. Instruction: for each, return JSON {index, coherent, label, keep_headings} where keep_headings = the member headings belonging to the dominant topic (empty if incoherent). Output a JSON array only.
• parse_reply(text) -> list[dict] — extract the JSON array from claude’s stdout (tolerate surrounding whitespace / code fences); return [{index, coherent, label, keep_headings}]; on parse failure raise (caller treats the whole batch as “keep raw” so a bad reply never silently drops clusters).

judge.py (run)

• apply_verdict(cluster, verdict) -> dict | None — pure: given a raw cluster + its verdict, return the refined cluster or None (drop). Matches keep_headings against member heading (exact, then normalized/trim fallback); recomputes source_manuals/sources/span/size; uses verdict label. Drop if coherent false or kept < 2.
• judge_all(clusters, batch_size=15) -> tuple[list, dict] — batch, call _run_claude(prompt), parse, map verdicts back by index, apply; return (surviving_sorted_reassigned, stats). A batch whose claude call fails or won’t parse → its clusters pass through UNCHANGED (fail-safe: never drop on error).
• _run_claude(prompt) -> str — subprocess claude -p --model opus, prompt on stdin, timeout 300, raise on rc!=0/empty.
• main_judge(batch_size) — load clusters.json, snapshot to clusters.raw.json, judge_all, write clusters.json, print stats (raw N, dropped, refined, kept, relabeled).

kbclust.py

• judge [--batch N] subcommand → judge.main_judge.

Failure / ops notes

• Fail-safe on claude errors/parse-fail: the affected batch’s clusters pass through UNCHANGED (better to keep a grab-bag than silently lose real clusters). Stats report how many batches fell back.
• clusters.raw.json snapshot before overwrite → always recoverable; re-run kbclust build regenerates raw from scratch.
• Serialized + thread-capped; ~13 opus calls one-time on the power-flaky box → run as one background job; re-runnable.
• Read-only over Qdrant + KB docs; only writes clusters.json / clusters.raw.json.
• Refinement may drop a cluster below span≥2 (e.g. two Vitara manuals = 1 source) — that’s fine, it’s still a real overlap; the explorer already shows span and sorts by it.
• Idempotent-ish: re-judging an already-judged clusters.json re-snapshots and re-judges (labels already clean → stable).

Testing / verification

1. judgeprompt unit: prompt lists each cluster with index + members; parse_reply extracts the JSON array (with/without code fences); parse failure raises.
2. apply_verdict unit: coherent+keep_headings → refined cluster with recomputed span/source_manuals/size + new label; keep<2 → None; coherent false → None; heading match handles exact + trimmed.
3. fail-safe: a batch with an unparseable claude reply leaves those clusters unchanged (simulate by feeding parse_reply junk → judge_all keeps them).
4. Real judge run: run kbclust judge on the live clusters.json; assert the window-regulator grab-bag (cluster containing “Filling and painting” + “WINDOW REGULATOR”) is GONE or reduced to a coherent remainder; assert a known-good cluster (parking brake / engine cooling) survives with a clean (typo-free) label; report raw→kept counts + a few before/after labels.
5. Explorer: /semantics still 200 and now shows clean labels, no window-regulator grab-bag; a surviving cluster’s member links still resolve.
6. Recoverable: clusters.raw.json exists and equals the pre-judge output.

Risks

Risk	Mitigation
claude drops a genuinely-good cluster	fail-safe keeps-on-error; raw snapshot for recovery; human can eyeball /semantics
Over-pruning (drops real members)	keep<2 drop is conservative; raw.json recoverable; spot-checked in testing
heading-match misses (OCR whitespace)	exact then trimmed/normalized fallback in apply_verdict
claude JSON malformed	parse_reply raises → batch passes through unchanged, reported in stats
~13 opus calls destabilise the box	single serialized background job, thread-capped, re-runnable
Judged labels drift from source	label is display-only; members still link to the authoritative KB sections