Levandor

Indexing the Obsidian vault into Qdrant — project-level selection + incremental sync

3 min read

How the notes Qdrant collection is built from the Obsidian vault — which notes are in scope, why scaffolding pages are dropped, and how the indexer stays incremental and resumable. Code lives in kb-vectors/obsidian_index.py; the search side is 2026-08-31-telep-kb-mcp-server.

Selection rules

Arrived at after several iterations. The unit of selection is a project, not a file:

  • Each directory under projects/
  • Each top-level directory in the vault
  • All vault-root .md files together, as one pseudo-unit

Then:

  1. A unit is excluded entirely if ANY note in it carries the frontmatter tag personal. Opt-out, not opt-in — one private note takes its whole project out of the index.
  2. Within an included unit, LOG.md and TOPICS.md are skipped as scaffolding.
  3. index.md, moc-tagged project overview pages, and Agent Landing.md are KEPT.

personal on any note removes the entire project from search

The tag is evaluated per-unit, not per-file. Tagging one note personal inside projects/<x>/ silently drops every note in <x> from the notes collection. If a project stops appearing in agent search results, grep its folder for the tag before debugging the indexer.

Current scope: 28 units excluded, 16 included, 218 notes, ~979 chunks.

Why LOG.md / TOPICS.md are dropped but index.md is kept

Both are generated index pages — near-pure wikilink lists. Homelab’s pair alone was 159 chunks at ~0.9 link density. They embed to noise and, being long, actively displace real answers in the top-k. Their content is derived restatement of notes that are already indexed individually, so nothing is lost.

Genuine prose pages measured 0.09–0.34 link density — an order of magnitude apart from the scaffolding.

A link-density heuristic was tried and rejected

Thresholding on link density looks principled and fails in practice: it split identical file roles incoherently — tatabanya TOPICS.md at 0.42 was kept while esp32 TOPICS.md at 0.67 was dropped. The metric tracks project size (a big project’s TOPICS has proportionally more prose around its links), not page quality. Exclude by filename/role, not by a measured proxy for it, when the role is already knowable.

Incremental and resumable

The indexer never calls delete_collection. Per note it stores a sha256 content_hash in the Qdrant payload and:

  • unchanged note → skipped, no embedding
  • changed note → re-embedded and upserted immediately, not batched to the end of the run
  • note deleted, or newly excluded → its points are deleted

Immediate upsert is the property that makes it resumable: a run killed halfway (which happens — see 2026-08-31-nvidia-drm-host-crash-embedding-pass) leaves every already-processed note durably indexed, and the next run picks up exactly where it stopped.

Verified live:

RunResult
1embedded 2
2re-embedded 0 (hashes matched)
3re-embedded only the edited note
4purged the deleted note’s points

Current state

The full index build is INCOMPLETE

Only 27 points across 5 of 218 notes are in notes. The full embedding pass hard-crashes the host — see 2026-08-31-nvidia-drm-host-crash-embedding-pass. telep-kb-obsidian-sync.timer is disabled for the same reason. Agent searches against notes will look mostly empty until that is resolved; this is not a retrieval bug.

Graph View

Backlinks