Knowledgebase vectorization — phase 1 design

Embed every OCR’d manual section into a Qdrant vector DB with rich metadata, and expose semantic search. Foundation for later semantic dedup (phase 2) and consolidated claude -p generation (phase 3).

The bigger picture (roadmap — phases 2/3 are NOT this spec)

Vectorize (this spec): chunk + embed the manual sections into Qdrant with metadata; provide semantic search.
Semantic dedup / cluster: group chunks that mean the same thing across manuals (the manuals overlap heavily). Uses the vectors from phase 1.
Consolidated generation: given a topic (“replace the differential”), retrieve its cluster across manuals, and have claude -p synthesize a merged markdown (with the relevant images) that highlights where sources differ.

Dropped for now: a suzuki-forums.com crawler (the site is Tollbit-gated against AI crawlers; not built).

Phase 1 goal

All manual content embedded in Qdrant, re-runnable as manuals are added, with a working semantic search (query -> ranked chunks + source metadata + images). Immediate side-benefit: much better search than the knowledgebase’s lexical mkdocs search.

Non-goals (phase 1)

Dedup, clustering, generation (phases 2/3).
A polished search UI (a CLI + small query function is enough; UI can come later).
Auto-indexing on upload (phase 1 is a standalone re-runnable indexer; hook into ingest later).
Forum/web sources.

Key decisions

Decision	Choice
Embeddings	`BAAI/bge-large-en-v1.5` (local, sentence-transformers), 1024-dim, cosine
Compute	GPU (RTX 3080) when free; falls back to CPU (both fine at this scale)
Vector store	Qdrant (docker container on the mainframe)
Source	`~/knowledgebase/manuals-src/docs/<folder>/<manual>/*.md` (the KB sections)
Chunk unit	section page, sub-chunked to fit the 512-token embed limit, with overlap
Metadata	manual, folder, page URL, heading, image list, source text
Idempotency	deterministic point IDs; re-index replaces a manual’s points
Runs as	`levander` (GPU + files reachable without root; Qdrant via docker)

Architecture

Qdrant (docker)                          collection: manuals (1024-dim, cosine)
~/kb-vectors/  (levander, venv)
  embed.py        load bge-large-en-v1.5 (sentence-transformers); embed_texts(), embed_query()
  chunker.py      pure: read a manual dir -> list[Chunk] (text, images, heading, page_url, token-bounded)
  index.py        (re)index one/all manuals into Qdrant (delete+upsert per manual)
  search.py       query -> embed -> Qdrant search -> ranked hits with payload
  kbvec.py        CLI: `index [--manual F/M | --all]`, `search "<query>"`, `stats`
  requirements    sentence-transformers, qdrant-client

Data flow

index: for each manual dir, chunker produces token-bounded chunks with metadata; embed vectorizes them; index deletes that manual’s existing points (by a manual payload filter) and upserts the new ones. Deterministic IDs = hash(folder, manual, page, chunk_index).
search: embed the query (with bge’s query instruction prefix), Qdrant top-k with optional folder/manual payload filter, return {score, manual, page_url, heading, images, text}.

Components

Qdrant

Docker container, restart: unless-stopped, persistent volume ~/kb-vectors/qdrant-storage, bound to 127.0.0.1:6333 (REST) / 127.0.0.1:6334 (gRPC). Tailnet-only if its UI is exposed later (optional dashboard tile at :6333/dashboard).
Collection manuals: size 1024, distance Cosine.

chunker.py (pure, testable)

chunk_manual(manual_dir, folder, manual) -> list[dict].
Read each *.md section (sorted). Strip the leading #/## heading for the heading field; keep it in the text.
Extract image refs ![](_page_*.jpeg) in the section → images (relative names; resolvable at <folder>/<manual>/<img> on the KB and on disk).
Sub-chunk the section text into windows of ~400 tokens with ~60-token overlap (bge-large max = 512). Token counting via the embed model’s tokenizer (or a char/word proxy if the tokenizer is heavy). Each chunk carries the section’s images (section-level association is enough for v1; phase-3 generation can pick per-procedure images), heading, and page_url = <folder>/<manual>/<page-stem>/.
Skip empty/near-empty chunks.

embed.py

Lazy-load SentenceTransformer("BAAI/bge-large-en-v1.5"), device = cuda if available else cpu.
embed_passages(texts) -> list[vec] (no prefix for bge passages).
embed_query(text) -> vec with bge’s query instruction: "Represent this sentence for searching relevant passages: " + text.
Normalize embeddings (cosine).

index.py

ensure_collection(client) — create manuals (1024, cosine) if missing.
index_manual(client, folder, manual) — chunk → embed → delete existing points where payload manual == "<folder>/<manual>" → upsert new points (id, vector, payload).
index_all(client, docs_dir) — iterate every <folder>/<manual>.

search.py

search(client, query, limit=8, folder=None, manual=None) -> list[hit] — embed_query → Qdrant search with optional payload filter → return payload + score.

kbvec.py (CLI)

python kbvec.py index --all / --manual suzuki-vitara/5door-supplement
python kbvec.py search "how to bleed the brakes" [--folder suzuki-vitara]
python kbvec.py stats — collection point count, per-manual counts.

Failure / operational notes

GPU contention: embedding is a short job; run it when the OCR queue is idle, or on CPU. Never blocks the OCR/knowledgebase.
Re-runnable & idempotent: re-indexing a manual replaces its points (no duplicates). Safe to run repeatedly as manuals are added by the OCR queue.
Qdrant persistence survives restarts (volume). Losing it just means re-index (cheap).
Read-only over the knowledgebase docs — cannot harm the live KB.
Model download: bge-large (~1.3GB) fetched once from HuggingFace to the local cache.
The box’s mains instability applies (Qdrant is another container that’s down during outages); acceptable, it just re-serves on boot.

Testing / verification

chunker unit tests: a section with images → chunks carry the images + heading + page_url; a long section → multiple token-bounded chunks with overlap; empty → no chunks; image refs parsed correctly.
embed check: embeds a few strings → 1024-dim normalized vectors; query vs passage prefixing differs.
Qdrant up: collection created (1024/cosine); stats shows a non-zero count after indexing.
index idempotency: index a manual twice → point count stable (no duplicates).
search quality (real): index the current manuals, query a known procedure (e.g. “brake bleeding”, “torque specification”) → top hits come from the right manual/section, and the payload carries the page URL + images. This is the real proof.
filter: a folder=suzuki-vitara search returns only Vitara chunks.
Re-run after the OCR queue adds a manual → new manual’s chunks appear; existing unchanged.

Risks

Risk	Mitigation
Chunks exceed bge 512-token limit (truncation)	token-bounded sub-chunking, verified against the tokenizer
Duplicate points on re-index	delete-by-manual-filter before upsert; deterministic IDs
GPU contention with OCR/Frigate	short job; run when idle or on CPU
Section-level image association too coarse	acceptable v1; phase-3 can refine per-procedure
Qdrant is another service on a contended box	lightweight; tailnet/localhost only; restceptable

Levandor

Explorer

Knowledgebase vectorization (phase 1) — design