Levandor

KB orphan-fix subsystem — deterministic heading recovery + gated re-embed/re-tag

3 min read

The subsystem that fixes the orphaned-procedure pages diagnosed in 2026-08-06-kb-marker-chunker-bug-and-ocr-bakeoff: it finds orphans, recovers the true component heading from marker’s RAW output deterministically (no re-OCR), gates uncertain fixes, then re-embeds and re-tags the affected Qdrant points. Reversible by design. On telep-mainframe, code under ~/knowledgebase/kb-vectors/.

For Agents — the pipeline in one place

  • orphans.py → kb-vectors/orphans.json — detects orphaned procedure pages (generic H1, no component name). 235 found.
  • pagesrc.py — locates a page’s body inside marker RAW (~/ocr/marker_out/<manual>/<manual>.md) and walks up to the nearest non-generic heading = the recovered component name.
  • orphanfix.py — recovers the heading, applies a confidence gate, and re-embeds + re-tags the page.
  • orphan_apply.py — batch driver over all orphans.
  • pageflow.py — the per-page operation flow: Inspect / Fix-heading / Re-OCR / Apply / Rebuild.

Result

  • 235 orphans found, of which 89 applied and 146 staged-uncertain (gated — see below).
  • 149 Qdrant points re-embedded + re-tagged.
  • Example: point 259 (IAC valve procedure) was mistagged Cooling → corrected to Fuel after the heading Idle Air Control (IAC) valve was re-attached and the text re-embedded.

The confidence gate — why 146 are staged, not applied

A WRONG component name is worse than a generic one

Heading recovery only auto-applies when confident. The 146 staged-uncertain cases are deliberately not written, because attaching the wrong component name (e.g. from an ambiguous raw-file region) actively misleads search — strictly worse than leaving a generic Removal and installation. Uncertain fixes wait for human review via the pageflow.py Inspect step (2026-08-06-kb-tools-ui-and-job-queue).

Re-embed + re-tag mechanics

The recovered heading changes a page’s text, so its vector and payload must be rebuilt:

  • Re-embed = new text → bge embed_passages → update the existing Qdrant vector + text payload in place, preserving all other payload keys. Same point ID, so nothing downstream breaks.
  • Re-tag = re-run the system-tag classifier via kb-vectors/tagger.py (see 2026-08-06-kb-embedding-classifiers-system-tags-and-image-types) so the corrected text lands in the right primary_tag/tags (that’s how point 259 moved Cooling → Fuel).

Model: bge-large-en-v1.5, Qdrant collection manuals (same as 2026-07-24-kb-vectorize-complete).

Reversibility

Every change is undoable

  • .bak-orphanfix backups of the modified source pages.
  • staging/orphan-changelog.json — a full changelog of what was changed (and what’s staged-uncertain), so any application can be reviewed or rolled back.

Graph View

Backlinks