KB wiring-diagram extraction (hybrid CV + vision) — design

Extract digitally-usable data from the scanned wiring diagrams in the car-repair knowledgebase: (1) searchable labels/text from every diagram (so diagram-only pages stop being invisible to search) and (2) best-effort structured connection data (a netlist) with per-item confidence flags, using a hybrid classical-CV path-tracing + vision-model pipeline. Works on the existing scans (no rescanning); the scan stays the source of truth.

Related: 2026-07-24-knowledgebase, 2026-07-24-kb-vectorize-complete, pipeline, telep-mainframe, SESSION-HANDOVER

Problem (root-caused via pilot)

Wiring diagrams live in the KB only as scanned raster JPEGs embedded in OCR’d markdown (marker/surya OCR’d the surrounding text but never the drawing). Consequences: diagram-heavy pages have almost no extracted text → they look empty (e.g. the geo-tracker cover) and are invisible to the vector search; and there is zero structured connectivity data (no netlist, pinout, or wire list).

Pilot finding (2026-07-29, Suzuki Vitara wiring-diagrams _page_14_Figure_0.jpeg, dense multi-system overview): vision extraction is PARTIAL — and the ceiling is scan resolution, not model capability.

• Labels/text: ~50–60 labels, titles ~95% / components ~85–90% confident → searchable-text target is viable now.
• Connections: topology traceable, but wire-color codes only ~60% (base letter reads, stripe letter fails), fuse amps / internal switch codes near-illegible.
• Hard constraint: the “full-res” source is only 1632×808 with ~9 systems per page → few pixels per label. No higher-DPI scan exists. Cropping/upscaling recovers structure but cannot invent unscanned detail.
• Judgment: single-circuit pages and/or 300–600 DPI rescans would push connections to ~90%+; on the existing dense overviews, connection data is inherently partial.

Goal

• Every wiring/diagram page gains a searchable text sidecar (labels, components, systems, decoded wire colors) that the existing vectorizer indexes → diagram pages become findable.
• A best-effort netlist per diagram (components, terminals, connections, wire colors) with confidence per item, emitted as JSON and in the existing WireViz “jot” line format so it round-trips into the wiring-diagram project’s renderer/BOM.
• The original scan remains authoritative; extracted connections are assistive, confidence-flagged, AI-marked — never presented as the manual.

Non-goals

• No rescanning in v1 (user chose best-effort on existing scans). A higher-DPI rescan path is a documented future accuracy lever, not built here.
• No full circuit simulation / SPICE. No editing/replacing the source scans or manual text.
• No new always-on service (box is power-flaky) — extraction is an offline batch job, orchestrated via Claude Code, output persisted.
• No interactive schematic viewer in v1 (possible later; the jot round-trip already yields renderable diagrams).

Key decisions

Decision	Choice
Targets	searchable labels (solid) + best-effort connections (confidence-flagged)
Connection tracing	hybrid: classical CV path-tracing for connectivity + vision model for semantics
Vision engine	Claude Code subagents (session model — box claude is weekly-limited); batch, not a service
CV stack	OpenCV + scikit-image (skeletonize) + numpy, CPU, thread-capped (OMP_NUM_THREADS=4, one job at a time — PSU limit)
Legend context	feed each manual’s own legend pages (color codes, symbols, connector conventions) to the vision passes
Fine text	crop-by-region + upscale (2–9×) before vision reads (pilot-proven)
Output	per-diagram JSON sidecar + WireViz jot lines + a vectorized searchable-text sidecar
Search integration	sidecar text, vectorized (source md/scan untouched; reversible; AI data kept separate from OCR)
Trust	scan = source of truth; AI-marked; per-item confidence; connections assistive; human review gate
Scope/phasing	Phase 1 = Suzuki Vitara wiring-diagrams; then suzuki-sidekick/wiring-1996, workshop electrical sections

Architecture

New module under /home/levander/knowledgebase/wiring/ (co-located with the app; batch CLI, no new service):

wiring/
  diagram_index.py   pure: find wiring/diagram images (wiring manuals + image-heavy low-text pages); load per-manual legend context
  cvtrace.py         classical CV path-tracing → connectivity graph (nets) + component/terminal candidate boxes
  vision_extract.py  orchestrate vision passes (labels, component+terminal localization, wire-color reads) with crop/upscale
  fuse.py            fuse CV nets + vision semantics → netlist with per-edge confidence; emit JSON + jot lines
  emit.py            write per-diagram JSON sidecar, jot file, and vectorized searchable-text sidecar
  wire.py            batch CLI: `index`, `trace <img>`, `extract <img>`, `run <manual>` (serialized, thread-capped)
  tests/             unit tests (stdlib unittest, matching KB convention)

CV path-tracing (cvtrace.py) — the connectivity engine

Classical pipeline; conductor geometry is deterministic and doesn’t hallucinate:

1. Preprocess: grayscale → adaptive threshold / Otsu → binary; despeckle (small connected-component removal).
2. Text/symbol suppression: remove text glyph blobs (connected components with text-like aspect/area) and component-symbol regions (from the vision component boxes) so only conductors remain — reduces line/label collisions.
3. Skeletonize: morphological thinning (skimage.morphology.skeletonize) → 1-px conductor lines.
4. Graph build: from the skeleton, endpoints = 1-neighbor pixels, junctions = ≥3-neighbor pixels; trace segments between nodes → an undirected graph (nodes: junctions/endpoints/terminals; edges: traced conductor runs).
5. Crossing vs connection: classify 4-way meetings as hop-over (no connect) vs junction (connect) using solder-dot detection (local blob at the meeting) + the manual’s convention (from the how-to-read legend); low-confidence where ambiguous.
6. Net extraction: union-find over conductor-connected nodes → electrical nets.
7. Terminal association: snap net endpoints to the nearest vision-located component terminal. Output: nets + edges + confidence (clean trace = high; broken/ambiguous/low-res = low).

Vision extraction (vision_extract.py) — the semantics engine

Per pilot, subagent-driven, fed the legend context + region crops:

• Component + terminal localization: detect component blocks, their names, and terminal/pin positions (bounding boxes) → feeds CV terminal association.
• Label/text sweep: all system titles, component names, notes → the searchable-text sidecar.
• Wire-color reads: read base/stripe codes near conductor endpoints (crop+upscale), decode via the legend color table; confidence per read.

Fusion (fuse.py)

Combine: CV net (edge geometry) + vision endpoints (component.terminal) + vision wire-color → connection = {from: comp.pin, to: comp.pin, wire_color: "<code> (<decoded>)", net_id, confidence}. Confidence = f(CV trace cleanliness, vision label confidence, color-read confidence). CV-confirmed edge + confident endpoints + confident color = high; any weak input downgrades. Emit netlist JSON + jot lines (<harness> <color> <from.pin> <to.pin> <purpose>).

Data flow

diagram JPEG + manual legend pages
  → cvtrace: binarize→skeletonize→graph→nets (connectivity, confidence)
  → vision_extract: components/terminals/labels/wire-colors (semantics, confidence)   [crop+upscale]
  → fuse: nets ∪ semantics → netlist {components, connections[conf], jot_lines} + label/text bundle
  → emit:
       docs/<...>/<page>.wiring.json     (structured netlist + labels + meta + source image ref)
       docs/<...>/<page>.wiring.jot      (WireViz jot lines)
       <sidecar text picked up by the vectorizer>   (labels + decoded colors → searchable)
  → (existing kbvec index run) diagram pages become searchable

Trust / accuracy guards

• Scan is source of truth: every sidecar links back to the source image; nothing overwrites the scan or the OCR’d md.
• Per-item confidence on every label, color, terminal, and connection; low-confidence items clearly marked.
• AI-extracted provenance on all output; connections are assistive, shown alongside the scan, never as the authoritative manual.
• Human review gate (reuse the KB draft→review pattern) before any extracted netlist is treated as trusted; reviewer eyeballs against the linked scan.
• No silent coverage caps: emit stats (pages processed, connections high/med/low, unreadable regions) so partial coverage is explicit.

Box constraints (bind the implementation)

• CV runs CPU-bound: OMP_NUM_THREADS=4 + nice, one heavy job at a time (PSU brown-out risk). No concurrent GPU+CPU heavy work (Frigate owns the GPU).
• Vision via Claude Code subagents (session model), not box claude -p (weekly-limited) and not a local VLM (GPU contention) in v1.
• Offline batch, re-runnable, idempotent per image; no always-on service.

Testing / verification

1. cvtrace unit: on a synthetic schematic (known lines + a dot-junction + a hop-crossing), assert correct net count, junction-vs-crossing classification, and endpoint detection.
2. fuse unit: given mock CV nets + mock vision terminals/colors, assert the netlist edges, decoded colors, and confidence downgrade rules.
3. diagram_index unit: correctly identifies wiring pages + loads the right legend context per manual.
4. CV-tracing spike (real): run cvtrace on the pilot page’s cranking sub-region; compare its net graph to the pilot’s hand-verified cranking topology — report net/edge recall + crossing errors. (Gate: if CV adds no connectivity signal over pure vision at this resolution, fall back to vision-only connections + flag it — do not ship dead complexity.)
5. End-to-end (real): wire run on the Vitara wiring-diagrams manual → per-page JSON+jot+text sidecars; spot-check the cranking netlist against the scan; confirm confidence flags are honest (no confident-but-wrong).
6. Search win: after a kbvec index, a query for a component only present in a diagram (e.g. “IC regulator”, “fuel cut controller”) now returns the diagram page.
7. No regression: existing KB tests pass; source md + scans byte-unchanged; clusters.json untouched.

Risks

Risk	Mitigation
CV tracing unreliable at 1632px density	spike-gated (test 4); fall back to vision-only connections + confidence flags; document rescan lever
Crossing-vs-junction misread (false nets)	solder-dot detection + legend convention + low-confidence marking; human review
Vision confident-but-wrong on colors/pins	per-item confidence, decode via legend, review gate, scan linked
Heavy CV/vision destabilises box	CPU thread-capped, serialized, one job at a time; batch offline
Extracted netlist mistaken for authoritative	AI-provenance + confidence + scan-as-source-of-truth + review gate
Scope creep (viewer, simulation)	v1 = sidecars + jot + search only; viewer/round-trip render are later

Future levers (not v1)

• 300–600 DPI rescans of key diagrams → connections ~90%+.
• Per-system single-circuit crops as canonical extraction units (5–10× pixels/component).
• Round-trip the jot output through WireViz → clean redrawn diagrams + BOM in the KB.
• Local VLM on the 3080 if subagent-driven volume becomes impractical.