Levandor

Knowledgebase — Built

7 min read

As-built “knowledgebase”: a tailnet-only manual library + OCR ingest portal, live at https://knowledgebase.taild4189d.ts.net as its own tailnet node (not a port on telep-mainframe). It renders OCR’d PDF manuals as an mkdocs-material static site (folder-tree browse, full-text search, dark mode, phone-responsive, lazy images) and a Flask+waitress portal (/upload) accepts a PDF, runs marker OCR on the GPU (reusing ~/ocr/venv), splits the markdown into per-section pages, copies images, and atomically rebuilds the site. Runs on the mainframe host (where the GPU + marker venv live); only the tailnet identity is separate.

Design in 2026-07-24-knowledgebase-design, task-by-task plan in 2026-07-24-knowledgebase-plan — this note is the completion record and the hard-won knowledge, not a restatement of either.

For Agents

URL: https://knowledgebase.taild4189d.ts.net — its own tailnet device, not a telep-mainframe port. Services (both enabled, survive reboot):

  • knowledgebase.service — Flask+waitress app on 127.0.0.1:8092, User=levander (NOT root).
  • tailscaled-knowledgebase.service — the dedicated second tailscaled that gives the knowledgebase node its identity.

Code: /home/levander/knowledgebase/ — split_manual.py, slugs.py, build.py, seed_import.py, ingest.py, app.py, mkdocs.yml, hooks.py + 26 unit tests (all pass). Own venv venv/ (flask, waitress, mkdocs-material). OCR: reuses the existing ~/ocr/venv marker pipeline as a subprocess (marker_single, TORCH_DEVICE=cuda) — no duplicate torch/model install. See scanned-pdf-to-markdown-marker. Library source: manuals-src/docs/<folder>/<name>/ (folder tree → mkdocs auto-nav); build output site/. Add manuals: the /upload portal (pick folder + name, PDF only), or seed_import.py to import already-OCR’d marker output without re-OCR. Seeded with: Suzuki Vitara 5-door supplement (59 pages / 1248 imgs) + Chevrolet Tracker 1995 owners manual (5 pages / 352 imgs). Since greatly expanded — see the Library expansion section below.

How it works (the reusable parts)

  • Chunked loading = split by # h1. Each manual’s marker markdown is split on its # h1 headings into many small per-section HTML pages (a single 795 KB markdown would otherwise be one huge page). Plus a mkdocs on_post_page hook (hooks.py) injects loading="lazy" into every <img> (verified 1600/1600 tags lazy). This is what fixed “Ulysses chokes on 1248 images”.
  • Browse = folder tree. manuals-src/docs/<folder>/<manual>/ → mkdocs auto-nav; no hand-maintained nav:. A new manual folder appears automatically after a rebuild.
  • One service serves BOTH the static site (at /) and the portal (/upload) — mkdocs output is static, so the Flask app hosts uploads and triggers rebuilds behind a single tailscale serve port.
  • Ingest = single background worker thread, GPU-serialised (one marker OCR at a time, the GPU is shared with Frigate). Job states queued → ocr → splitting → building → done/failed, polled by the job page.
  • Atomic build swap. mkdocs builds into a temp dir on the SAME filesystem, then os.rename swaps site/. A broken source never takes down the live site.

The two that cost thought (separate-node + TLS)

Making the app appear as its own tailnet device — without a container or GPU passthrough — is the whole trick here, and one TLS trap inside it burned real time.

A separate tailnet identity on one host = a second tailscaled with userspace networking

To give one app its own <name>.<tailnet>.ts.net device while it keeps running on the host (where the GPU + marker venv are), run a dedicated second tailscaled with --tun=userspace-networking, its own --socket and --statedir. Then, against that socket: tailscale --socket=… up --hostname=knowledgebase (one interactive auth, like exit-vpn was) and tailscale --socket=… serve …. Result: knowledgebase.<tailnet>.ts.net as its own node — the lightweight alternative to a GPU-passthrough Incus container with a duplicate ~5 GB OCR stack.

tailscale serve --https fails with "no TailscaleVarRoot" unless the second tailscaled has a --statedir

Start the second tailscaled with only --state=<file> and tailscale serve --https 443 fails with “no TailscaleVarRoot” — a TLS handshake error where curl gets 000 even with -k, while TCP and ping succeed (so it looks like a listener/firewall problem, but isn’t). Cause: with only a state file, tailscaled has no var root to store the Let’s Encrypt cert. Fix: start it with --statedir=<dir> instead (the state file lives inside the statedir, so the node stays authenticated). Then tailscale --socket=… cert <name> provisions the cert and TLS works.

Security & robustness

  • Path-traversal guard (slugs.safe_slug + a resolve_within realpath containment check) is applied before ANY filesystem write from an upload. Proven sound in review: sibling-dir, symlink, absolute-component and %2e%2e payloads are all rejected. The final destination is always inside manuals-src/docs/.
  • Uploads restricted to %PDF magic bytes (not just extension) and capped at MAX_CONTENT_LENGTH=500 MB (413 on oversize).
  • Re-ingest must rmtree the destination first. Re-ingesting the same folder+name without clearing the destination leaves stale orphan section pages/images that show up as ghost pages in nav/search. Fixed in both ingest.py and seed_import.py.
  • Atomic swap means an OCR or mkdocs-build failure never corrupts the live library or the served site/ — the job goes failed with the error tail, the old site stays up.
  • Runs unprivileged (levander); no secrets involved.

Library expansion (2026-07-24)

The library grew well past the two seed manuals. Many OCR’d manuals now live under docs/<folder>/<manual>/:

  • suzuki-vitara: 5door-supplement, supplement-61a40 (99501), workshop-1988-1998 (835 pg), wiring-diagrams (99512), body-measurements.
  • chevy-tracker: owners-1995, geo-tracker-repair.
  • suzuki-sidekick: wiring-1996, parts-catalogue-89-98.

Plus the kick-fix.com community archive under docs/kick-fix/ (25 topics / 127 articles). All of it is now semantically searchable via 2026-07-24-kb-vectorize-complete (2839 chunks in Qdrant).

OCR pipeline pattern + discipline

  • Pipeline: marker (~/ocr/venv, marker_single --force_ocr, TORCH_DEVICE=cuda) → seed_import.py (splits + copies images) → build_site(). See scanned-pdf-to-markdown-marker.
  • CHAINED nohup jobs to serialize the single GPU. Manuals were queued as chained nohup scripts, each waiting on the prior’s .done sentinel, so the one GPU does one OCR at a time (and coexists with Frigate). This is the manual-batch analogue of the app’s single-worker queue.

Always md5-dedup downloads BEFORE OCR — many were exact or same-content duplicates

A lot of the source downloads were exact md5 duplicates or same content under different names (workshop had 2 copies, 99501 had 2, parts-catalogue had 2). OCR is expensive and GPU-serialised — md5-dedup first so you don’t burn GPU hours re-OCRing the same PDF. Also: zips sometimes contain the PDF (the sidekick wiring download was a zip of wiring2.pdf) — unpack and check before queuing.

Ecosystem (this session)

The knowledgebase became its own tag:telep tailnet node (not just a serve port), and a sibling global dashboard node was added — see 2026-07-24-global-dashboard. Both use the same separate-tailscaled-identity pattern documented below.

Behaviours that are by design (not bugs)

  • A bare folder URL 404s. mkdocs section pages have no folder index, so …/5door-supplement/ returns 404 by design; the nav links to section pages (…/001-introduction/), not folders.
  • Coarse split when the OCR has few h1s. The # split is on the ORIGINAL doc’s h1 count: the Tracker OCR had only 4 h1s → 5 coarse pages (vs the Vitara’s 59). Coarse navigation, not a perf problem (images lazy-load). A finer split (by h2 or by size) is a possible later enhancement.

Current state / verification

  • Live and verified end-to-end over the tailnet: root 200, /upload 200, section pages 200, search index present, both seeded manuals in the nav, images serve, and a real portal upload ran ocr → done in ~20 s and rendered.
  • Both services active + enabled (survive reboot). All 26 unit tests pass.

This is on telep-mainframe, whose mains power is unstable

The knowledgebase (and its second tailscaled node) are down whenever the box is, until the UPS is sorted. See 2026-07-23-mains-power-shutdowns.

Graph View

Backlinks