Knowledgebase — manual library + OCR ingest portal — design

Knowledgebase — design

A tailnet-only “knowledgebase”: a browsable, searchable, phone-friendly renderer for OCR’d PDF manuals, plus a portal to ingest new PDFs (OCR → into the library).

Related: SESSION-HANDOVER, telep-mainframe, scanned-pdf-to-markdown-marker, filestash

Goal

Replace “open the giant markdown in Ulysses” (which chokes on 1,248 images) with a fast web service on the tailnet that lets you browse manuals by folder, full-text search across them, read page-by-page on a phone with lazy-loaded images, and add new manuals by uploading a PDF that gets OCR’d automatically.

Non-goals

• Public exposure. Tailnet-only, via tailscale serve.
• Editing manuals in the UI.
• Non-PDF ingest (“PDF only for now” — explicit).
• Auth/multi-user (tailnet is the boundary, matching the rig’s other services).
• Auto-categorisation (user picks the folder on upload).

Key decisions

Decision	Choice
Service name	knowledgebase (knowledgebase.service, site title “Knowledgebase”)
Renderer	mkdocs-material static site (search, responsive, dark mode)
Chunked loading	split each manual by section (# h1) → many small HTML pages + loading="lazy" images
Browse	folder tree under docs/ → mkdocs nav
Portal	same service hosts /upload; one service, one tailscale-serve port
Filing	user picks folder + name on upload
OCR	reuse the existing ~/ocr/venv marker pipeline (subprocess), GPU
Runs as	levander (no root)

Architecture

One service on telep-mainframe. mkdocs builds a static site; a small Flask app serves that site AND hosts the ingest portal and OCR job runner.

~/knowledgebase/
  venv/                      flask, waitress, mkdocs-material (+deps)
  app.py                     Flask: serve site/ at /, portal at /upload, job API
  ingest.py                  job: marker OCR -> split -> rebuild (background thread, GPU-serialised)
  split_manual.py            pure: split one marker .md into ordered per-section files
  slugs.py                   pure: safe slug/path validation (path-traversal guard)
  build.py                   run `mkdocs build` into a temp dir, atomically swap site/
  mkdocs.yml                 material theme, search, hooks
  hooks.py                   mkdocs hook: add loading="lazy" to <img>
  manuals-src/docs/          the library source tree (browsable structure)
    suzuki-vitara/5door-supplement/{01-*.md, ..., _page_*.jpeg}
    chevy-tracker/owners-1995/{...}
  site/                      mkdocs build output (served static)
  jobs/                      per-job staging + status json
knowledgebase.service        systemd unit (levander)
tailscale serve :8446 -> 127.0.0.1:8092

Request flow

• GET / … GET /<manual pages> → static files from site/ (mkdocs-material: nav, search, lazy images).
• GET /upload → the ingest form (PDF file; folder = datalist of existing folders + free text; manual name).
• POST /upload → validate (PDF magic bytes; slug-safe folder+name), stage the PDF, enqueue a job, redirect to /jobs/<id>.
• GET /jobs/<id> → status page that polls GET /api/jobs/<id>.
• GET /api/jobs/<id> → {state, step, detail} where state ∈ queued|ocr|splitting|building|done|failed.

Components

split_manual.py (pure, the reusable core)

split_manual(markdown, title) -> list[Section] where Section = {order:int, slug:str, heading:str, body:str}.

• Split on lines matching ^# (h1). Everything before the first h1 → a leading section 00-intro (only if non-empty).
• Each section file keeps its # heading and body verbatim, including ![](_page_N_*.jpeg) image refs (images live in the same folder, so relative refs resolve unchanged).
• order is a zero-padded sequence (01, 02, …) so mkdocs’ filename-sorted auto-nav preserves document order.
• slug = order prefix + a slugified heading, e.g. 03-engine-mechanical.
• Heading text becomes the nav title (mkdocs uses the first # as the page title).

write_sections(sections, dest_dir) writes NN-slug.md files; images are copied separately by the ingest step.

Why split: a single 795 KB markdown renders as one enormous HTML page — the opposite of “chunked loading”. Per-section files give many small pages that load independently; images inside each use loading="lazy".

slugs.py (pure, security-critical)

safe_slug(text) -> str — lowercased, [a-z0-9-] only, collapse dashes, trim, max length; raises ValueError on empty result. Used for BOTH the folder and the manual name before they touch the filesystem. This is the path-traversal guard: ../, absolute paths, and shell metacharacters cannot survive slugification. The final destination is always manuals-src/docs/<safe_folder>/<safe_name>/ and is verified to be inside docs/ (realpath prefix check) before any write.

ingest.py (background job, GPU-serialised)

A single worker thread + a queue; only one OCR job runs at a time (marker uses the GPU, shared with Frigate). Per job:

1. ocr — ~/ocr/venv/bin/marker_single <staged.pdf> --output_dir <job_tmp> --force_ocr with TORCH_DEVICE=cuda. Capture the marker stdout tail for progress/detail.
2. splitting — read <job_tmp>/<stem>/<stem>.md, split_manual(...), write_sections(...) into docs/<folder>/<name>/, and copy all _page_*.jpeg from the marker output into the same folder.
3. building — build.py (mkdocs build to a temp dir, atomic swap of site/).
4. done / failed (with the error tail). A failure never corrupts the existing library or the served site/ (staged writes, atomic swap).

build.py

mkdocs build --site-dir <tmp> then atomically replace site/ (rename swap). A broken source never takes down the currently-served site.

mkdocs.yml + hooks.py

• theme: material, features: navigation.instant, search enabled, dark/light toggle.
• Auto-nav from the docs/ directory tree (no hand-maintained nav:), so a new manual folder appears automatically after rebuild.
• hooks.py: an on_post_page hook that injects loading="lazy" into <img> tags (chunked image loading on phones). Optional: glightbox for tap-to-zoom diagrams (deferred; not required for v1).

app.py (Flask + waitress)

• Serves site/ statically at /.
• /upload GET/POST, /jobs/<id>, /api/jobs/<id>.
• Upload validation: content starts with %PDF; folder+name via safe_slug; reject otherwise with a clear message.
• Runs under waitress (pure-python WSGI) bound to 127.0.0.1:8092; tailscale serve maps a tailnet port to it.

Serving / exposure

The app must appear as its OWN tailnet device named knowledgebase (https://knowledgebase.<tailnet>.ts.net), NOT a port on the telep-mainframe node. It still runs on the mainframe host (the GPU + ~/ocr marker venv live there); only the tailnet identity is separate.

• A dedicated tailscaled instance on the host provides the separate node identity — the lightweight alternative to a GPU-passthrough container:
  • tailscaled --tun=userspace-networking --state=/var/lib/tailscale-kb/tailscaled.state --socket=/run/tailscale-kb/tailscaled.sock as tailscaled-knowledgebase.service.
  • tailscale --socket=/run/tailscale-kb/tailscaled.sock up --hostname=knowledgebase (one interactive auth, like exit-vpn was).
  • tailscale --socket=/run/tailscale-kb/tailscaled.sock serve --bg --https 443 http://127.0.0.1:8092.
  • Result: https://knowledgebase.<tailnet>.ts.net → the app.
• knowledgebase.service: Restart=always, User=levander, WorkingDirectory=~/knowledgebase, ExecStart=venv/bin/python app.py, binds 127.0.0.1:8092.
• Alternative (not chosen, heavier): a dedicated Incus knowledgebase container with its own tailscaled + GPU passthrough + a duplicate marker/torch install. Rejected to avoid duplicating the ~5GB OCR stack and GPU-passthrough setup.

Initial contents

• suzuki-vitara/5door-supplement/ from the already-OCR’d ~/ocr/marker_out/g16b_manual_specific/ (split on ingest-import).
• chevy-tracker/owners-1995/ from the Tracker owners OCR (running now).
• A one-off import path/script seeds these two from existing marker output without re-OCR (same split+copy as a job’s steps 2-3).

Failure handling

• Upload of a non-PDF / bad slug → rejected before any job starts.
• OCR failure → job failed with the marker error tail; library untouched.
• mkdocs build failure → served site/ unchanged (atomic swap only on success); job failed.
• Concurrent uploads → queued; one GPU job at a time.
• Service crash → Restart=always; in-flight job is lost (re-upload), but the library on disk is intact.
• GPU contention with Frigate: OCR is heavy but tolerated (as during the manual OCRs); acceptable for occasional ingests.

Security

• Tailnet-only (tailscale serve), no public exposure.
• Path traversal: folder/name are slugified and the destination is realpath-checked to be inside docs/ before any write. The single most important guard.
• Upload restricted to PDFs by magic-byte check (not just extension).
• Runs unprivileged (levander); no secrets involved.
• Note: this box’s mains power is unstable — the service (and the whole box) is down during outages until that’s addressed (UPS). Not this project’s concern, but stated.

Testing / verification

1. split_manual unit tests: front-matter-before-first-h1 → intro section; N h1 → N (+1) ordered sections; image refs preserved verbatim; ordering zero-padded; empty/edge inputs.
2. safe_slug unit tests: ../../etc → safe slug or ValueError; unicode/spaces → hyphens; empty → ValueError; the realpath containment check rejects a crafted traversal.
3. build.py: builds the seeded library; atomic swap leaves a valid site/; a deliberately broken source leaves the previous site/ intact.
4. Seed import: import the two existing manuals; confirm folders/pages/images render and search finds a known term.
5. End-to-end ingest: upload a small PDF via the portal → job progresses ocr→splitting→building→done → the new manual appears in the nav and search, with images.
6. Phone/browse: load over the tailnet on a phone; confirm responsive layout, folder nav, search, and that images lazy-load (not all at once).
7. Path-traversal attempt: POST a folder like ../../foo → rejected, nothing written outside docs/.

Risks

Risk	Mitigation
Path traversal via folder/name	slugify + realpath containment (tested)
One huge HTML page (no chunking)	split by section into many pages
Build breaks the live site	build to temp + atomic swap
GPU contention / two OCRs at once	single job worker, serialised
Non-PDF or malicious upload	magic-byte check, PDF-only
mkdocs auto-nav mis-orders pages	zero-padded numeric filename prefixes