Levandor

KB orphaned-procedure pages are a marker CHUNKER bug, not OCR — plus the OCR bake-off decision

4 min read

The KB’s “orphaned procedure-fragment” pages (generic H1s like Removal and installation with no component name) are a marker per-page SPLITTER failure, NOT an OCR-legibility problem — the true component headers survive verbatim in marker’s RAW output and are recoverable deterministically without re-OCR. A GOT-OCR2 bake-off confirmed OCR is not the bottleneck: full re-OCR helps only for catastrophically-mangled spec tables and regresses clean prose. DECISION: patch (deterministic header re-attach), not full re-OCR. On telep-mainframe; feeds the orphan-fix subsystem in 2026-08-06-kb-orphan-heading-recovery-subsystem.

For Agents — the headline finding

  • Symptom: many KB pages have a generic procedure H1 (# Removal and installation, # Inspection, # Disassembly) and no component name, so they’re unsearchable by the thing they actually document.
  • Root cause = marker’s per-page SPLITTER dropped the component header, not OCR. The real header is intact in marker’s RAW markdown: ~/ocr/marker_out/<manual>/<manual>.md (e.g. # Idle Air Control (IAC) valve sits just above the orphaned procedure body).
  • Therefore: recover deterministically from the raw file — no GPU re-OCR needed. The fix is a text lookup + heading re-attach, not a vision model.
  • Bake-off verdict below: patch, don’t re-OCR. Targeted micro-re-OCR only for genuinely destroyed spec tables + code-block pages.

Why it’s a splitter bug, not OCR

The KB splits each manual’s marker markdown into per-section pages on # H1 boundaries (see How it works (the reusable parts)). When marker emits a page-break between a component header and the procedure that belongs to it, the splitter starts a new page at the generic procedure H1 — the component name is stranded on the previous page (or folded into unrelated content). The OCR text itself is fine; the document structure is what broke.

Proof: the component header is present, correctly spelled, in the raw marker output:

~/ocr/marker_out/<manual>/<manual>.md
    ...
    # Idle Air Control (IAC) valve      ← survives in RAW
    ## Removal and installation          ← this became the orphan page's generic H1
    ...

Because the header survives in the raw file, recovery is a deterministic string operation (locate the orphan’s page body in the raw markdown, walk up to the nearest non-generic heading) — implemented in 2026-08-06-kb-orphan-heading-recovery-subsystem.

The OCR bake-off — GOT-OCR2 vs the marker baseline

To be sure re-OCR wasn’t the better lever, a bake-off ran GOT-OCR2 (stepfun-ai/GOT-OCR-2.0-hf, ~3.6 GB VRAM, ~10–24 s/page) against marker’s existing output.

Where GOT-OCR2 wins:

  • Recovers dropped headers (but so does the deterministic raw-file lookup, for free).
  • Rebuilds catastrophically-mangled spec tables — e.g. kickfix-docs/specsgeotracker1994/004-power-teams, where marker produced garbage.

Where GOT-OCR2 regresses (why NOT a full re-OCR):

  • Regresses clean prose — introduces character noise on pages marker already handled well.
  • Emits LaTeX, not GFM — tables/markup come out as LaTeX, wrong for the Markdown pipeline.
  • Fragile page seams — inconsistent boundaries between pages.
  • Useless on wiring schematics — line-art diagrams are not text.

DECISION — patch, do NOT full-re-OCR

Use the deterministic header re-attach for the orphan problem (the real win, no GPU cost). Reserve targeted micro-re-OCR (GOT-OCR2) only for the small set of destroyed spec-table pages and code-block pages where marker’s text is genuinely unrecoverable. A blanket re-OCR would trade a fixable structural bug for widespread prose regression.

SOTA OCR landscape (as of 2026-08)

Noted for future evaluation, not adopted:

  • GLM-OCR
  • DeepSeek-OCR 2
  • Dolphin — structure-first (parses layout before text; relevant to the splitter problem)
  • PaddleOCR-VL

Two OCR gotchas discovered

  • JustVugg/colibri is NOT an OCR model — it’s an MoE-LLM inference engine. Do not add it to the OCR shortlist.
  • DeepSeek-OCR will not install on this box — it needs flash_attn and transformers 4.4x, but the box has transformers 5.x installed. Would require a dependency downgrade / isolated env.

Key paths

Where the recoverable headers live

  • Source PDFs: ~/ocr/*.pdf
  • marker RAW output: ~/ocr/marker_out/<manual>/<manual>.md (+ *_meta.json)
  • GPU venv for any re-OCR: ~/gpuvenv (torch cu124)
  • KB content (split pages): ~/knowledgebase/manuals-src/docs

Graph View

Backlinks