Levandor

Knowledgebase vectorization (phase 1) — Built

3 min read

As-built completion record for phase 1 of knowledgebase vectorization: every OCR’d manual section (plus the kick-fix community articles) is now embedded into a local Qdrant vector DB with per-chunk metadata and images, and semantic search works across all sources. Result: 2839 chunks indexed across all 33 manual/kick-fix collections, with cross-source semantic search that blends official manuals and community guides. This is the foundation for phase 2 (semantic dedup) and phase 3 (claude -p consolidated generation).

Design in 2026-07-24-kb-vectorize-design, task-by-task plan in 2026-07-24-kb-vectorize-plan — this note is the completion record and the hard-won knowledge, not a restatement of either.

For Agents

Code: /home/levander/kb-vectors/ on telep-mainframe (own venv, runs as levander). Modules: chunker.py, embed.py, index.py, search.py, kbvec.py (CLI: index / search / stats). Embeddings: local BAAI/bge-large-en-v1.5, CPU torch + sentence-transformers, 1024-dim, cosine, normalized. Passages: no prefix. Queries: prefix "Represent this sentence for searching relevant passages: ". Vector store: Qdrant docker container kb-qdrant, bound 127.0.0.1:6333 (REST) / :6334 (gRPC), storage ~/kb-vectors/qdrant-storage, collection manuals (1024-dim cosine). qdrant-client 1.18. Chunker reuses the KB’s per-section pages (word-windowed, carries images + metadata). Idempotent per-manual index: delete-by-manual-filter + deterministic uuid5 point ids → re-index replaces, never duplicates. State: 2839 chunks across 33 manual/kick-fix collections; cross-source search verified.

Why these choices

  • CPU torch, not GPU — sidesteps the CUDA-13-vs-driver-550 trap and avoids GPU contention with the OCR queue + Frigate. Embedding a few thousand chunks on the 12900K is fine (with the thread cap below).
  • Qdrant over an embedded store — payload filtering (search within a manual), a built-in UI, and room to grow into phases 2/3.
  • Reuse the knowledgebase’s per-section pages as the chunk source rather than re-parsing PDFs — the KB already split, cleaned and image-linked everything.

Verified cross-source behaviour

Semantic search blends official manuals + kick-fix community guides well:

  • "depressurize the fuel line" → kick-fix/injectors de-pressurize (0.79) + vitara Engine Fuel.
  • "oxygen sensor testing" → geo-tracker + kick-fix/sensors.

This is already noticeably better than the knowledgebase’s lexical mkdocs search.

Hard-won: this triggered the PSU-under-load resets

Default 16-thread CPU embedding maxed all cores and crash-reset the whole box

Running the full index with bge’s default thread count (all 16 cores) reset telep-mainframe within 2–6 minutes, every time. This is a PSU-can’t-sustain-peak-load fault, NOT thermal and NOT mains — see 2026-07-24-psu-load-resets for the full diagnosis and the router-uptime discriminator.

Cap embedding to 4 threads + nice, and serialize heavy jobs

OMP_NUM_THREADS=4 / torch.set_num_threads(4) + nice, and run heavy jobs one at a time (not GPU OCR + CPU embedding + kick-fix conversion + Frigate all at once). With the cap the box stayed up 45+ minutes and completed the full 2839-chunk index. The real hardware fix is a bigger/better PSU.

Scope notes

  • A suzuki-forums.com crawler was DROPPED — the site is Tollbit-gated against AI crawlers.
  • Phases 2 (semantic dedup) and 3 (claude -p consolidated generation) are NOT done — this is the vector foundation they build on. See the roadmap in 2026-07-24-kb-vectorize-design.

Graph View

Backlinks