Box is UP — booted 2026-07-24 13:55 after someone physically pressed power following a morning mains outage. clevis TPM auto-unlock worked again (unattended encrypted-root unlock). All services active: knowledgebase, home-portal, wifi-usage, intruder-alarm, camwall, tailscaled-knowledgebase, tailscaled-home; kb-qdrant + frigate containers Up (frigate.service shows inactive because Frigate is docker, not systemd — expected). Qdrant manuals collection: 2839 points. A resume OCR/vectorize job is in-flight (see “In-flight” below) — check it finished.
Since last handover (2026-07-30)
Two KB milestones landed: the Hungarian translation of the manuals is complete for the prioritized
set, and the AI wiring-netlist extraction was retired in favour of a curated scan gallery.
KB Hungarian translation — prioritized corpus COMPLETE (820/820)
What. The KB manual markdown is now translated to Hungarian for the full prioritized set:
820/820 pages (956 total .hu.md incl. consolidated). Manuals: chevy-tracker/geo-tracker-repair
434, suzuki-vitara/workshop-1988-1998 293, 5door-supplement 61, supplement-61a40 23,
suzuki-sidekick/wiring-1996 9, all kick-fix/* how-tos, consolidated. Deliberately skipped
(low value, addable later): parts-catalogue-89-98, body-measurements, wiring-diagrams booklet,
owners-1995, kickfix-docs/*.
Serving. Read-only at /hu/<page_url> via mkdocs-static-i18nsuffix mode (English fallback for
untranslated). NOT indexed for search — search stays English (bge-large-en only).
Method (crash-safe). Per-page .hu.md siblings written by session-model FORK subagents over
disjoint page ranges, each page flushed to disk the INSTANT it’s translated
(heredoc/ssh-stdin/scp piping to survive quotes/$/backticks) + grep-verified. ⚠️ The first bulk
attempt lost the entire run to a mid-run API drop (nothing flushed); after switching to flush-per-page,
3 API “connection closed” drops AND a hard session usage-limit cost ZERO completed work — re-run,
skip-if-.hu.md-exists resumes. ~6–12k tokens/page.
Faithfulness contract. Translate prose+headings; keep VERBATIM all numbers/units/torque/clearance/
resistance/voltage/part-numbers/DTC/wire-colors/mileage/URLs + every  ref; tables have
labels translated + numeric cells verbatim; don’t translate garbled OCR tables; lookup/paint-mark CODES
stay English; banner > AI-forditas. A hiteles forras az angol eredeti. on every page.
⚠️ Two open judgment calls: page 198 bearing paint-mark colors were translated (Green→Zöld) — revert
if they’re codes; page 202 one garbled OCR TOC cell dropped. Full detail
2026-07-30-kb-hungarian-translation-complete.
Why. The AI netlist extraction (2026-07-29-kb-wiring-extraction-v1-complete) was deemed too
inaccurate for agents (scan-resolution ceiling). Retired it, replaced with the accurate ground truth.
Retired (reversibly, nothing deleted).hooks._append_wiring no longer called (on-page netlist
tables + Cytoscape graphs gone); 65 .wires.txt sidecars moved to /home/levander/wiring-retired/;
wiring-charts.md + write_charts_index disabled; the 2 wiring manuals reindexed so
Qdrant//api/search returns no extracted netlist text (marker count 0). Backups kept.
Replaced with /wiring-gallery/. A consolidated scan library: 480 curated genuine wiring-diagram
images (auto-included the 5 dedicated wiring manuals + vision-verified the ambiguous ones, dropping
147 photos/icons/exploded-views), 320px thumbnails, grouped vehicle→manual, each thumbnail links to
the full source scan. Home-page nav link + cited in kb-agent-api as cluzter’s tracing-underlay
source. selection.json + thumbs at /home/levander/wiring-collection/.
glightbox zoom. Enabled mkdocs-glightbox (click-to-zoom, full-screen pan/zoom) → fixed the
“diagrams look cut off” issue (wide diagrams were shrunk into the ~60em content column).
Distinction for agents. Retirement kills the AI netlists, NOT the OCR’d scans (which remain
good and are cluzter’s underlay). Agents: go to the scan/gallery, never /api/search, for wiring
connectivity. Full detail 2026-07-30-kb-wiring-gallery-pivot-complete.
Since last handover (2026-07-29)
A new KB semantic-search frontend shipped (the KB’s new front door); KB wiring-diagram extraction v1
built, then scaled to real content + a graphical render layer; plus a camwall latency fix.
KB semantic-search frontend v1 SHIPPED — replaces the slow lunr search as the KB front door
What & why. mkdocs-material’s client-side lunr search hung on “Initializing search” (a
5.4 MB / 6683-section browser-built index) and is unfixable in-config: community Material
removed prebuild_index (Insiders-only), and dropping the HU search lang either doubles the index
(reconfigure_search:false → 13366 docs) or removes the HU site. So we replaced it with a server-side
SEMANTIC search page as the primary way in.
Where it runs. IN the existing knowledgebase Flask app (waitress @127.0.0.1:8092, Restart=always)
— no new service. The semantic backend already existed (2026-07-24-kb-vectorize-complete:
kb-vectors/search.py + embed.py bge-large over Qdrant manuals); the frontend just exposes it.
Installed CPU-torch 2.13.0+cpu (NO CUDA) + sentence-transformers 5.6.1 + qdrant-client 1.18.0 +
transformers 5.14.1 + numpy 2.5.1 into the app venv so search.py imports in-process (~1.63 GB
RSS of 64 GB). app.py does sys.path.insert(0,"/home/levander/kb-vectors"), holds module global
QCLIENT=QdrantClient(127.0.0.1:6333), and main() warms the embedder once + torch.set_num_threads(4)
before serving (lazy singleton → import app stays cheap).
Routes.GET /api/search?q=&folder=&manual=&limit= → JSON
{took_ms,count,hits:[{heading,manual_id,vehicle,page_url,snippet,score,type}]} (dedupe-by-page +
snippet/highlight [HTML-escaped, XSS-safe] + type classify + vehicle label). / + /search →
search-first UI (dark, autofocus id=kb-search, debounced fetch, type badges manual/wiring/article,
vehicle filter, ↑/↓/Enter nav, i18n hu+en). /browse → the mkdocs index (kept reachable). Catch-all
/<path:path> still serves site/; site() is no longer decorated with /. New searchui.py
(pure helpers + tests); backups app.py.bak-search-fe / app.py.bak-search-ui.
Verified live. Warm query ~54 ms end-to-end. Real semantic wins: “why won’t the fuel pump prime”
→ “Pump is dead, now what” (0.728), “fuel cut controller” → FUEL CUT SYSTEM (0.726), “stop light switch”
→ Stop Light Switch Adjustment (0.681). Folder filter works; wiring hits deep-link to
#kinyert-huzalozasi-adatok-ai. 141 tests pass. ✅ Graceful degradation: Qdrant stopped →
/api/search 503 but home + manual pages still 200, service didn’t crash (NRestarts=0); restored →
search back. Non-destructive (Qdrant/docs/site//clusters untouched). v1 = retrieval only, NOT RAG
synthesis. Phase-2 candidates: answer synthesis via claude (cited), hybrid semantic+BM25 fusion, and
retire the lunr search on /browse entirely. Full detail
2026-07-29-kb-semantic-search-frontend-complete.
Scaled to real content. Beyond the thin wiring-diagrams booklet (10/10), the full
suzuki-vitara/5door-supplement manual is now 55/55 diagram pages processed — extracted, rendered
on-page, vectorized/searchable. Qdrant carries ~377 points for this manual; 55 pages carry the
“Diagram data (extracted)” marker. Finding confirmed at scale: clean single-system circuits
(stop/tail/interior lights, wipers, defogger, washers, DTC sensor circuits 037–043) trace well at
high/med confidence — clearly beating the ~60% dense-booklet ceiling; bundled harness / operation diagrams
(central-locking, power-window, the 060 body-electrical multi-market page) stay honestly conservative
(med/low, power-path + representative edges); routing/flowchart/block/mechanical are labels-only — no
fabrication. Honest gap: page 044’s crank-angle-sensor “circuit” is only a flowchart+pinout in the
scan (not drawn), so left labels-only.
On-page render hook — hooks.pyon_page_markdown (merged with the kickfix deeplink rewriter +
lazy-image) appends a per-page ”## Kinyert huzalozási adatok (AI)” section from the per-page JSON
(/home/levander/wiring-extract/out/<folder>__<manual>__<page>.wiring.json): an HTML connections table
(Honnan|Huzalszín|Hová|Bizt.) + components + systems + AI-provenance note. Non-destructive (source md
untouched; reads the <page>.wires.txt / JSON sidecar at render time).
Mermaid graphical renderer (NEW) — enabled pymdownx.superfences + a mermaid custom fence in
mkdocs.yml (mkdocs-material 9.7.7, client-side). The hook also emits a per-page ”### Áramköri gráf (AI)”mermaid graph LR: nodes=components, edges labeled with decoded wire color, LOW-confidence edges dashed
(-.->), grouped into a subgraph per subsystem, ~30-edge/subgraph readability cap. Applies to all 62+
processed pages (render-layer, no re-extraction). ⚠️ Markup + Material’s mermaid loader bundle verified,
but NOT a live-browser render (Chrome extension wasn’t connected; Material lazy-loads mermaid from CDN →
needs browser internet). Live-browser confirm is an open follow-up.
Folder/manual-root 404 fix — gen_index.py extended to generate a landing index.md for every folder
manual dir (idempotent <!-- generated-index --> marker), so /suzuki-vitara/5door-supplement/ etc.
(which 404’d under use_directory_urls) now list their pages. Backup gen_index.py.bak-404.
Empty pages are OCR artifacts, not bugs — divider/cover/TOC pages (e.g. 017-wiring-diagram = heading
only; the geo-tracker cover) render fine but are thin by nature; the thin booklet’s real schematics were
concentrated on page 019 (marker grouped 6 figures there).
Reusable box tooling in /home/levander/wiring-extract/: cvtrace.py (masked-CV: surya text-mask +
vision symbol suppress_boxes → skeletonize → dot-vs-crossing → union-find nets), legend.py/legend.json
(13 Suzuki wire colors), diagram_index.py, emit.py, run.py (status/reindex/search), per-page spec
drivers emit_driver.py/mech_emit.py/build_page.py + specs/. Vision+fusion is subagent-driven
(box claude weekly-limited). Next extraction targets: workshop-manual electrical sections + Sidekick
wiring-1996. Full detail 2026-07-29-kb-wiring-extraction-v1-complete.
The 2×2 wall ran ~10–13 s behind live; isolated to mpv’s single-composite movie= lavfi buffering
(camera + go2rtc were each only ~1 s). Fixed by rearchitecting /usr/local/bin/camwall-mpv.sh into
4 tiled single-input low-latency mpv instances (per-quadrant --geometry, --profile=low-latency --cache=no, per-pane drawtext labels) → ~1–2 s per pane, stable. Backup camwall-mpv.sh.bak-latency.
⚠️ Two known regressions (open follow-ups): (a) the WiFi strip is now confined to the BL pane
(was full-width) — cosmetic; (b) watchdog freeze-detection now only covers the one IPC-socket-owning pane
(cam2/BL), not all 4 — security-relevant, fix via per-pane IPC sockets or framebuffer-based per-quadrant
staleness. Full detail Latency fix (2026-07-29) — 4 tiled single-input mpv instances.
⭐ SUPERSEDED / REAL ROOT CAUSE (2026-07-31): the residual multi-second camwall lag is 2.4 GHz camera-WiFi
bufferbloat, NOT an mpv/go2rtc problem. The wandering per-pane lag (up to ~12 s, worst on DÉL) that
survived even a full rebuild to a go2rtc MSE/chromium wall is the 2.4 GHz telep-cc radio at 98.9%
channel-busy (ping to both cameras 350–420 ms / ~700 ms peaks / 0% loss = bufferbloat; noise floor clean
-106 dBm so it’s self-congestion from the cameras’ own uplink). Driven by 4 continuous full-HD Frigate
record streams all over one 2.4 GHz channel. No player-side fix can resolve it. Ranked fixes:
(1) get cameras off 2.4 GHz — wire Ethernet/PoE (best) or 5 GHz; (2) cut record bitrate/res (camera-side
changes hard remotely: pytapo has no encoder setter, ONVIF flaky); (3) changing WiFi channel WON’T help.
Note the separate inherent ~2 s substream keyframe/GOP floor. Full evidence chain + fixes:
2026-07-31-camwall-lag-24ghz-wifi-bufferbloat.
New module /home/levander/wiring-extract/ on the box (house-rules-clean, no comments): legend.py+legend.json (13 Suzuki wire colors transcribed from scan _page_8_Picture_7.jpeg, decode_color("B/Y")->"Black/Yellow", symbols/connector rules/abbreviations; any venv), diagram_index.py (list_diagram_pages() — 10/21 Vitara wiring pages are diagram pages, 38 images), cvtrace.py the connectivity engine (runs under ocr/venv: cv2 4.11 + scikit-image 0.26 [installed this session] + surya; trace(image, suppress_boxes, crop, debug) = Otsu binarize → despeckle → TEXT mask via surya DetectionPredictor → SYMBOL mask via vision-supplied boxes → skeletonize → graph → dot-vs-crossing classifier → union-find nets; bridge=1 gap-bridge), emit.py (.wiring.json + <page>.wires.txt + .lines), run.py (status/reindex/search/run_page), fuse_pilot.py (per-page vision extraction+fusion template).
kb-vectors chunker.py PATCHED (backup chunker.py.bak-wiring): merges a page’s <page>.wires.txt onto that page’s chunks (same page_url); the sidecar never becomes its own page. 51 kb-vectors tests pass. NON-DESTRUCTIVE — source md / scans / clusters.jsonmd5-unchanged.
Proven end-to-end on 1 pilot page (suzuki-vitara/wiring-diagrams/019-light-head-…): it went from 0 extracted text (unfindable) to #1 within its manual for “fuel cut controller” / “IC regulator” via the indexed sidecar. Coverage 1/10 Vitara diagram pages — remainder is pure scale.
Naive CV path-tracing is a NO-GO on these 1632px scans — mask text + symbols first
Skeletonizing the raw binary → 161 false junctions / 502 segments because TEXT (titles, labels, fuse ratings) and COMPONENT SYMBOLS (fuse coils, switch boxes) skeletonize into dense false-junction clusters. Line-following worked; connectivity didn’t. Masking text (surya auto-detect ~35 boxes) + symbols (vision boxes) before skeletonizing → junctions 161 → 37 (-77%), conductors intact. Symbols were the bigger contaminant (161 → 106 text-only → 37). Division of labor: vision locates/reads → boxes to cvtrace as suppress_boxes → cvtrace returns nets → vision fuses semantics + reconnects sub-pixel gaps, never fabricating untraceable connections. Full detail 2026-07-29-kb-wiring-extraction-v1-complete.
⚠️ Scan resolution is the accuracy ceiling (1632×808 @ ~9 systems/page; wire-color stripe letters B/R vs B/Bl ~60%) — 300–600 DPI rescans or single-circuit pages would push connections to ~90%+ (future lever).
⚠️ Scaling is token-heavy: vision+fusion is Claude Code subagent-driven (session model, NOT box claude -p/weekly-limited, NOT a local VLM/GPU contention) at ~13 min + ~110k tokens PER PAGE. To do a page: run vision extraction+fusion (template fuse_pilot.py) → emit.write → then OMP_NUM_THREADS=4 nice /home/levander/kb-vectors/venv/bin/python run.py reindex suzuki-vitara wiring-diagrams.
Since last handover (2026-07-28)
Network detour + a second camera + the KB Hungarian-toggle brainstorm.
macOS “slow internet” root-caused to broken IPv6 (FIXED)
The Mac’s browsing was intermittently “absolutely ass” while the wired box + raw bandwidth were fine (300/38–48 Mbit, 18–20 ms, 0% loss). Root cause:telep-router advertised a global IPv6 prefix (2a00:1110:210:7653::/64) + ULA (fd83:5e9d:98ca::/64) via RA/DHCPv6, but IPv6 had no working upstream (fixed-wireless + Double-NAT). macOS prefers v6 (RFC 6724 / Happy Eyeballs) → tried the dead path first → 2–7 s stall per dual-stack site → IPv4 fallback.
Fix (applied + verified): on the router uci set dhcp.lan.ra/dhcpv6/ndp='disabled' + network.wan6.disabled='1', commit, odhcpd restart + network reload. Mac connect times 2–7 s → 0.02–0.06 s. Backups /etc/config/dhcp.bak-ipv6fix + network.bak-ipv6fix, reversible. Full detail 2026-07-28-ipv6-slow-internet.
⚠️ Do not re-enable LAN IPv6 until the upstream carries it. The WAN is a TP-Link NE200 outdoor fixed-wireless unit, double-NAT’d (WAN got 192.168.254.2 from the NE200 at 192.168.254.1) — bridging/DMZ-ing the NE200 to kill the double-NAT is the separate fix that could restore real IPv6.
net-monitor link-quality logger installed
/home/levander/net-monitor/probe.sh + user crontab (every 5 min) → /home/levander/net-monitor/netlog.csv (timestamp_iso,rtt_avg_ms,rtt_max_ms,loss_pct,dl_mbps,loaded_rtt_ms). Pings 1.1.1.1 each run; a 20 MB throughput+bufferbloat sample only near the top of the hour (~480 MB/day). Purpose: characterize suspected fixed-wireless peak-hour degradation. Review column -s, -t netlog.csv | tail; remove crontab -l | grep -v net-monitor/probe.sh | crontab -. Full detail 2026-07-28-net-monitor.
Second camera (ÉSZAK) + camwall moved OFF birdseye
Added a second Tapo TC47 ÉSZAK (192.168.30.139, telep_cam3 fix + telep_cam4 ptz) to Frigate alongside DÉL (telep_cam1/telep_cam2). Now 4 cameras / 4 go2rtc substreams (cam1_sub..cam4_sub).
⚠️ Frigate birdseye is UNRELIABLE as a fixed >2-cam wall source — it flip-flops between showing 2 and 4 cameras. camwall-mpv.sh rewritten to composite the 4 substreams directly into a grouped 2×2 (DÉL left, ÉSZAK right) via an mpv --vf lavfi filtergraph (cam1_sub primary + cam2/3/4 via movie=), 4 drawtext labels, TOP KÉPEK overlay dropped, wifi-usage IPC strip kept (/run/camwall.sock). Keep --hwdec=no. Backup camwall-mpv.sh.bak-2x2. Full detail 2026-07-28-camwall-4-substream-composite.
⚠️ Gotcha: mpv movie= rtsp URLs need colons escaped (rtsp\://...\:8554/...); movie= can’t take rtsp_transport (benign 461 Unsupported transport, falls back UDP→TCP); if go2rtc restarts a movie= pane can freeze until camwall restart (watchdog only covers the primary input).
telep_cam3 (ÉSZAK – Fix, aimed indoors, false-triggering) excluded from Telegram alerts. Correct key:frigate.cameras.exclude: [telep_cam3] (nested UNDER the frigate: block, by Frigate camera id) — NOT alerts.cameras.block, NOT a top-level cameras:. frigate-notify v0.5.4, config /home/levander/nvr/frigate-notify/config.yml. ⚠️ koanf silently ignores misplaced keys — a first attempt at top level logged “Config validated! / App ready!” and still sent the alert. Verify against a real event (Event dropped - Camera Excluded camera=telep_cam3), never config validation. Re-enable: remove block + docker restart frigate-notify. Backup config.yml.bak-cam3excl. Full detail 2026-07-28-frigate-notify-camera-exclude.
User chose Approach A: mkdocs-static-i18n + Material language selector for the static site, + a shared lang cookie for the Flask dynamic pages; HU/EN toggle covering UI chrome + cluster labels + generated consolidated articles; source manuals stay English.
Design NOT yet written to a spec doc — paused pending the network detour above. Resume via the brainstorming → writing-plans flow.
Since last handover (2026-07-24)
Delta since the last handover commit a197813.
Router Tailscale upgraded 1.80.3 → 1.98.9
telep-router tailscale hand-upgraded to a static arm64 1.98.9 build (the opkg feed only carries 1.80.3). Router online, 8 peers, verified over SSH served by the new binary.
Old binaries backed up at /overlay/ts-backup/ (persists across reboot). Done via a detached self-reverting swap script that rolls back to the backup if the new binary isn’t online within 45s.
setsid, not nohup, for a self-reverting service swap on OpenWrt/busybox
busybox has no nohup — use setsid to detach a swap script that could otherwise kill your only remote foothold. Keep a persisted binary backup on /overlay (survives reboot) and auto-revert on failure (roll back if the new service isn’t online within N seconds), so a bad swap can’t lock you out of the router.
The router tailscale binary is out-of-band — opkg may clobber it
The 1.98.9 binary was hand-installed; opkg still RECORDS 1.80.3-r1. A future opkg upgrade would silently restore 1.80.3 over the new binary. Re-apply the static build (or hold the package) after any opkg upgrade. Backup lives at /overlay/ts-backup/.
Daily ~06:00 mains die-off (NEW observation — a mains signature, not PSU)
The box appears to lose power around 06:00 most mornings — deaths clustered Jul 19 ~06:59, Jul 23 ~06:36, Jul 24 morning. These take the router down too (router uptime resets with the box) → MAINS-level, distinct from the PSU-load resets (where the router survives 14h+).
Suspected: a scheduled utility event — a Hungarian off-peak/controlled (vezérelt/GEO) tariff relay switching a controlled circuit ~06:00, OR a timed high-draw appliance (electric bojler/heater) on the same circuit browning out the feed. NOT confirmed.
To confirm next time the box is up: journalctl --list-boots + correlate every death with router reboots for clock-regularity.
Daily ~06:00 mains die-off is UNCONFIRMED
Three mornings of ~06:00 power loss that take the router down too (= mains, not PSU). Cause not yet proven — suspect a scheduled utility relay (vezérelt/GEO tariff) or a timed high-draw appliance on the shared circuit. Confirm with journalctl --list-boots vs router reboot times for clock-regularity. SEPARATE from the PSU-under-load resets (the router SURVIVES those — see 2026-07-24-psu-load-resets).
Remote-wake (WOL) is NOT possible as configured
After a mains outage the box needs a physical power-button press. WOL was never enabled (BIOS + NIC); the router has no etherwake/wakeonlan; and after ~6h off the box’s MAC is unknown (DHCP lease expired, no static reservation, ARP cleared).
To make remote-wake possible later: (a) BIOS enable Wake-on-LAN, (b) persistent ethtool -s enp5s0 wol g, (c) a static DHCP reservation on the router so the MAC is always known, (d) a WOL tool on the router.
Better fix: BIOS Restore on AC Power Loss → Power On so the box auto-recovers from the daily mains blip like the router already does (needs one physical BIOS visit).
In-flight — check it finished next session
~/ocr/resume_extras.sh (setsid-detached; sentinels ~/ocr/resume_extras.done / .failed): finishes the interrupted extras OCR. body-measurements already OCR’d + seeded into suzuki-vitara; this OCRs the Sidekick parts-catalogue (sidekick_parts_89_98.pdf) → seeds suzuki-sidekick/parts-catalogue-89-98 → rebuilds the site → then vectorizes BOTH new manuals (body-measurements + parts-catalogue) into Qdrant (thread-capped OMP_NUM_THREADS=4 + nice, per the PSU lesson → will push manuals past 2839 points).
If it died (another power event), re-runbash ~/ocr/resume_extras.sh.
Access (do this first)
macOS gotcha: bare ssh is a broken shell wrapper. Always use /usr/bin/ssh.
Mainframe: /usr/bin/ssh levander@telep-mainframe (tailnet) or …@192.168.1.123 (LAN). Passwordless sudo.
Router: /usr/bin/ssh root@telep-router (OpenWrt, busybox — no timeout, classic ip). NEW: a LAN-only, key-only dropbear now also listens on 192.168.1.1:22 (added so the mainframe can read WiFi presence). GOTCHA: enabling it rotated the router’s SSH host key → the Mac’s known_hosts entry for telep-router is now stale (Host key verification failed); remove the offending line to reconnect.
Mainframe→router (for the alarm): /usr/bin/ssh -i /home/levander/.ssh/router_alarm root@192.168.1.1.
Agent containers from Mac: /usr/bin/ssh alpiq (also facekom, crypto, alpiq-test, facekom-test) via ProxyCommand through mainframe incus.
Secrets live on the Mac in ~/telep/ (all 600). NOT on the mainframe (the alarm reads the bot token from ~/nvr/frigate-notify/config.yml).
Double-NAT behind 192.168.254.1. WAN outbound is intermittently flaky (causes ~15s Telegram send latency + occasional dig/API failures — not the code, the uplink).
⚠️ Recurring “turns off overnight” = MAINS POWER loss, not the PC (2026-07-23). Confirmed mains: the telep-router (separate PSU) died at the same instant (mainframe journal dead 06:36:10, router booted 06:36:27). Journal stops dead with no Stopping/panic/thermal trace; temps cool; no poweroff timers. The PC stays dark only because BIOS Restore on AC Power Loss = stay-off (the router auto-powers-on, so its recovery is invisible). Fix: a ~600–900 VA UPS + NUT (logs sags, graceful shutdown); free stopgap = flip BIOS AC-loss to Power On (physical BIOS access only, not over SSH); also rule out an overloaded shared circuit (i9+3080+TV). ext4 clean so far but repeated hard cuts risk corruption. Full detail in 2026-07-23-mains-power-shutdowns.
⚠️ SECOND, distinct power problem (NEW 2026-07-24): PSU-can’t-sustain-peak-load resets. The box crash-rebooted ~3× in 30 min (04:11/04:20/04:32) while running MANY heavy jobs at once (GPU OCR + CPU embedding + kick-fix conversion + Frigate). NOT thermal (CPU 39°C/GPU 47°C, no MCE), NOT mains — the router stayed up 14h+ straight through (the router-uptime cross-check is the discriminator: mains = router dies too, PSU = only the PC resets). Dead-stop journal signature. Root cause: the PSU can’t hold peak combined i9-12900K + RTX 3080 draw; specifically the default 16-thread CPU embedding maxing all cores reset the box in 2–6 min every time. A UPS does NOT fix this — that’s for (a) mains. Fix (b): serialize heavy jobs + cap threads (OMP_NUM_THREADS=4/torch.set_num_threads(4) + nice → box stayed up 45+ min, completed the full index); real fix = a bigger/better PSU. Full detail in 2026-07-24-psu-load-resets.
⚠️ THIRD signature — a daily ~06:00 mains die-off (NEW 2026-07-24, UNCONFIRMED). The mains outages (a) cluster around 06:00 most mornings (Jul 19 ~06:59, Jul 23 ~06:36, Jul 24 morning) and take the router down too (= mains, not PSU). Suspected: a scheduled utility relay (Hungarian vezérelt/GEO off-peak tariff) or a timed high-draw appliance browning the shared circuit — not proven. Confirm via journalctl --list-boots vs router reboot regularity. Remote-wake is impossible as configured (WOL off, MAC unknown after lease expiry) — the real fix is BIOS Restore on AC Power Loss → Power On. Full delta in the “Since last handover (2026-07-24)” block above + 2026-07-23-mains-power-shutdowns.
Drives (2026-07-21): ⚠️ NVMe device nodes flip between reboots — always lsblk -f, never hardcode. Currently nvme1n1 = 3.6T OS disk (p1 /boot/efi, p2 /boot ext4, p3 LUKS+LVM); nvme0n1 = 1.8T Windows NTFS disk, C: on nvme0n1p2 (~886G used). sda 447G unused (single 16M part, no fs). sdb 115.5G = Hiren’s BootCD PE USB stick (UDF, HBCD_PE_x64).
Disk encryption / auto-unlock (NEW 2026-07-24 — full detail in 2026-07-24-luks-tpm-autounlock): root is LUKS2 on LVM (container /dev/nvme0n1p3, mapper nvme1n1p3_crypt, UUID a3a8e37d-79fa-484b-bc3f-40c56df95337). Was passphrase-only → sat at an unlock prompt after every reboot (unreachable over SSH, no dropbear-initramfs). Now has clevis TPM2 auto-unlock: keyslot 1 = TPM (bound with empty PCRs, sudo clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'), keyslot 0 = passphrase (fallback, untouched). TPM /dev/tpmrm0 (MSFT0101), Secure Boot OFF. Check:clevis luks list -d /dev/nvme0n1p3. Header backed up first at ~/telep/luks-header-nvme0n1p3-20260723.img (600, on the Mac). ✅ Reboot test PASSED IN PRODUCTION 2026-07-24 — the box auto-unlocked root through a 3-reboot PSU-load storm, no passphrase, all services self-recovered (see 2026-07-24-psu-load-resets). ⚠️ Auto-unlock only helps once the box boots — BIOS Restore on AC Power Loss is still stay-off, so a mains cut still needs a button press (see 2026-07-23-mains-power-shutdowns). Empty-PCR + Secure-Boot-off tradeoff: protects against bare-drive theft, NOT whole-box theft.
Frigate NVR (~/nvr/ on mainframe)
docker compose: frigate (0.17.2, ONNX yolov9-s @640 on GPU, NOT tensorrt) + frigate-notify (polls API 15s, pushes Telegram, no MQTT). Media /srv/frigate.
⚠️ UPDATED 2026-07-28: now TWO dual-lens Tapo TC47s.DÉL @ 192.168.30.119 (telep_cam1 fix / telep_cam2 ptz) + ÉSZAK @ 192.168.30.139 (telep_cam3 fix / telep_cam4 ptz) on the isolated telep-cc VLAN. 4 go2rtc substreams cam1_sub..cam4_sub. telep_cam3 is aimed indoors and excluded from frigate-notify alerts (frigate.cameras.exclude, see 2026-07-28-frigate-notify-camera-exclude). Per lens: /stream1+/stream2 fixed, /stream6+/stream7 PTZ. Tracks person+car, face_recognition on.
Telegram bot “Dezsi az őr” @tlphlyortbot → group -1004475187307. Alerts on person+car (no retry — WAN blip = lost alert).
Frigate viewer alert (NEW — full detail in 2026-07-23-frigate-viewer-alert):frigate-viewer-alert.service (runs as levander, NOT root; enabled, survives reboot) follows docker logs -f --since 0m frigate, reads the real client IP from each nginx access line’s X-Forwarded-For (set by tailscale serve), maps 100.x → tailscale device name, and Telegrams 👁 <device> (100.x) opened Frigate — HH:MM[ · live: <cam>] when someone opens the UI. Reuses the frigate-notify bot+chat (token/chatid read at runtime from ~/nvr/frigate-notify/config.yml, never copied) so viewer pings land in the same chat as person/car alerts. Internal automation has XFF - → excluded by construction. Per-source-IP 5-min session cooldown (an open tab polls constantly → edge-trigger). Code /home/levander/frigate-viewer-alert/ (python3 stdlib, 34 tests). Only unverified leg = a real browser session (can’t be faked into the live stream) — user 3-step manual test in the note.
TV wall — “camwall” (X11 + NVIDIA)
⚠️ UPDATED 2026-07-28: the cctv pane no longer plays birdseye.camwall-mpv.sh now composites the 4 go2rtc substreams into a grouped 2×2 (DÉL left, ÉSZAK right) via mpv --vf lavfi, because birdseye flip-flops between 2 and 4 cameras once you have >2 cams. See the “Since last handover (2026-07-28)” block + 2026-07-28-camwall-4-substream-composite.
camwall-mpv.sh now uses --hwdec=no (software decode). ⚠️ The RTX 3080’s NVDEC was corrupting the live feeds → whole wall went magenta/green; software decode on the 12900K fixed it (GPU contention w/ Frigate detection was the cause). Debug it by grabbing the X framebuffer: sudo DISPLAY=:0 XAUTHORITY=<xorg -auth> ffmpeg -f x11grab -i :0.0 ….
Pane system:pane-runner.sh reads /srv/frigate/pane (cctv|web), pane-switch <pane> [url] writes state + pkills current app. cctv → camwall-mpv.sh composite; web → chromium kiosk.
WiFi usage strip (NEW — full detail in 2026-07-22-wifi-usage-strip):wifi-usage.service (root, enabled, survives reboot) polls the router every 5s over one persistent SSH connection, diffs iw station dump byte counters into live Mbps, and pushes an ASS osd-overlay into the running mpv. Renders a strip at the bottom of the cctv pane (y 1880–2140, below TOP KÉPEK) with every telep1 device’s ↓/↑ plus a 5G WAN total header; 7 tiles, then “6 busiest + +N további”. Code /home/levander/wifi-usage/ (python3 stdlib, 44 tests). Names reused from the alarm’s ~/alarm/trusted.json → DHCP hostname → eszköz-<last4>. Router failure keeps last values with ⚠, then — after 60s — never blanks the wall.
Reload wall images without restart:camwall-reload (mpv IPC loadfile). ⚠️ It reloads through the current --vf — do NOT put a filter in the vf that can fail to configure (e.g. pad with a mismatched size), it kills the whole composite → raw birdseye only. Filter changes need systemctl restart camwall, not camwall-reload.
Top-képek picker (~/top4-web.py, service top4-web, ts serve :8443)
Slot-replace UI. NEW: in-browser CROP — arm a slot, click a detection → crop modal (free-form, all-8 handles, drag to move). The crop is letterboxed to exactly 880×495 in the browser canvas (black bars for non-16:9) and posted as a finished JPEG. This keeps the wall’s scale=880:495 filter simple/unbreakable (the pad-in-wall-filter approach crashed the wall — see camwall gotcha). New endpoints /full/<id>, /set-slot-image, /export.
NEW: “Exportálás Drive-ra” button → /export runs top4-export. top4-export now writes gdrive:top_képek/<date>_<HH-MM-SS>/ (timestamped, no overwrite). top4-export.timer Mon 08:00 unchanged.
Printer (NEW)
HP LaserJet M203dw @ 192.168.1.126 (NPIB8AE8C), driverless IPP, set as CUPS default on the mainframe. lp file.pdf just works locally.
Remote access: router advertises subnet route 192.168.1.126/32 (approved). Mac: add by IP (ipp://192.168.1.126:631/ipp/print, IPP-Everywhere). iPhone AirPrint over Tailscale is IMPOSSIBLE (mDNS doesn’t cross Tailscale; iOS print sheet is discovery-only) — use HP Smart / a direct-IP app instead.
What: tailnet-only manual library + OCR ingest portal. mkdocs-material static site (folder-tree browse, full-text search, dark mode, phone-responsive, lazy images) of OCR’d PDF manuals, plus a Flask /upload portal that OCRs a PDF and adds it to the library.
Its OWN tailnet node:https://knowledgebase.taild4189d.ts.net — NOT a port on the telep-mainframe node. Runs on the mainframe host (GPU + marker venv live there); only the tailnet identity is separate, via a dedicated second tailscaled.
Two systemd units (both enabled, survive reboot):knowledgebase.service (Flask+waitress app on 127.0.0.1:8092, User=levander not root) and tailscaled-knowledgebase.service (the dedicated node’s tailscaled, --tun=userspace-networking, own --socket/--statedir).
Ingest reuses the ~/ocr marker venv as a subprocess (marker_single, TORCH_DEVICE=cuda) — no duplicate torch/model install (see scanned-pdf-to-markdown-marker). Single background worker, GPU-serialised (shared with Frigate); job states queued→ocr→splitting→building→done/failed. mkdocs builds to a temp dir then os.rename-swaps site/ (a broken source never takes down the live site).
Add manuals: the /upload portal (pick folder + name; PDF magic-byte check, 500 MB cap) or seed_import.py (import already-OCR’d marker output without re-OCR). Library source manuals-src/docs/<folder>/<name>/ → mkdocs auto-nav.
Library now GREATLY expanded (2026-07-24):suzuki-vitara/ (5door-supplement, supplement-61a40/99501, workshop-1988-1998 835pg, wiring-diagrams/99512, body-measurements), chevy-tracker/ (owners-1995, geo-tracker-repair), suzuki-sidekick/ (wiring-1996, parts-catalogue-89-98). OCR pipeline: marker (~/ocr/venv, marker_single --force_ocr, TORCH_DEVICE=cuda) → seed_import.py → build. Manuals queued as CHAINED nohup scripts (each waits on the prior’s .done sentinel) to serialize the single GPU. ⚠️ md5-dedup downloads BEFORE OCR — many were exact dupes or same-content different-names (workshop ×2, 99501 ×2, parts-catalogue ×2); zips sometimes contain the PDF (sidekick wiring was a zip of wiring2.pdf). Full detail 2026-07-24-knowledgebase.
kick-fix.com archive ingest (NEW — full detail 2026-07-24-kickfix-ingest): 112 MB offline archive of the dead kick-fix.com Suzuki repair site → docs/kick-fix/ (25 topics, 127 articles, 741 images, 8 text-layer PDFs; 27 image-only PDFs deferred to OCR). Dead-site archive = no ToS/robots issue. Script /home/levander/knowledgebase/kickfix_ingest.py (KB venv + markdownify + beautifulsoup4 + pypdfium2). See gotchas.
Phase 1 DONE: every manual section + kick-fix article embedded into a local Qdrant vector DB. 2839 chunks across 33 manual/kick-fix collections; cross-source semantic search works and blends official manuals + community guides (e.g. “depressurize the fuel line” → kick-fix injectors 0.79 + vitara Engine Fuel).
Code/home/levander/kb-vectors/ (own venv, runs as levander): chunker.py, embed.py, index.py, search.py, kbvec.py (CLI: index/search/stats). Embeddings: local BAAI/bge-large-en-v1.5, CPU torch, 1024-dim cosine, normalized; query prefix "Represent this sentence for searching relevant passages: ". Store: Qdrant container kb-qdrant on 127.0.0.1:6333/:6334, storage ~/kb-vectors/qdrant-storage, collection manuals. qdrant-client 1.18. Idempotent per-manual (delete-by-manual-filter + deterministic uuid5 ids).
⚠️ The 16-thread CPU embedding is what reset the box (see PSU-load bullet above) — run capped to 4 threads + serialized. Suzuki-forums crawler DROPPED (Tollbit-gated against AI crawlers).
Phase 2 DONE: fully-local semantic SECTION clustering over the phase-1 vectors + a read-only web explorer, NO LLM. Pipeline: scroll all 3252 chunk vectors from Qdrant manuals (READ-ONLY) → group by (manual_id,page) → L2-normalized mean-pool → ~1248 SECTION vectors → HDBSCAN(min_cluster_size=2, min_samples=1, euclidean on unit-norm ≈ cosine) → representative-heading labels → clusters.json. Result: 194 clusters (post boilerplate filter), 24 cross-SOURCE (span≥2), 631 noise singletons (noise is EXPECTED/correct for a dedup goal). Top clusters are real cross-make overlaps (WINDOW REGULATOR spanning chevy-tracker + suzuki-vitara/5door + workshop; ENGINE COOLING, PARKING BRAKE, THROTTLE BODY).
Code/home/levander/kb-vectors/ (reuses the phase-1 venv, runs as levander): new sectionvecs.py, labels.py, cluster.py, kbclust.py (+ tests). CLI:kbclust.py build/list/show <id>; build regenerates clusters.json on demand (thread-capped per the PSU lesson) after the corpus changes.
Explorer:GET /semantics + GET /api/clusters folded into the EXISTING knowledgebase Flask app (/home/levander/knowledgebase/app.py) — NO new always-on service on the power-flaky box. Live at https://knowledgebase.taild4189d.ts.net/semantics (Hungarian “Szemantika”); each cluster shows label + N források + members linking into the KB pages; cross-source first; empty state if clusters.json missing. Dashboard tile 🧭 Szemantika on home.taild4189d.ts.net.
Key decisions: section-level unit (not chunk); local heading labels pluggable behind make_label (a claude -p relabel drops in later); ranked by cross-SOURCE span so dedup targets surface first.
✅ Phase 3 (claude -p consolidated generation) is now DONE — see the section below. clusters.json was indeed its retrieval bundle. Optional bridge (a claude -p relabel pass behind make_label) still available but not built.
Phase 3 DONE — the full knowledge system is now COMPLETE end-to-end (OCR ingest → vectorize → semantic clustering → on-demand cited consolidation). On-demand consolidated repair-article generation from a cross-source cluster via claude -p --model opus, in a draft → human review → publish workflow. Proven live.
Flow:/semantics cluster “Generál” button → POST /generate/<id> → single serialized background worker gathers the cluster’s member section .md + images → rewrites image refs to source-unique names (see ⭐ gotcha) → never-invent prompt → claude -p --model opus (prompt on stdin) → writes a DRAFT (drafts/<slug>/ = draft.md + meta.json + images) → runs speccheck → review at /drafts/<slug> (AI-provenance banner + speccheck warnings + Közzététel/Elvetés buttons) → publish moves it to docs/consolidated/<slug>/ (build_site atomic swap, browsable, vectorizable) or discard deletes it.
Code/home/levander/knowledgebase/ (runs as levander): genprompt.py (pure prompt build), speccheck.py (pure number-token check), gen.py (background job), kbgen.py (CLI: cluster/list-drafts/publish/discard) + tests. Co-located in knowledgebase/ NOT kb-vectors/ (design’s proposed home) — deliberately, to avoid cross-venv imports + reuse the KB app’s build_site/slugs.safe_slug/docs/ access. Routes added to the existing KB Flask app.py: POST /generate/<id>, GET /drafts, GET /drafts/<slug>, POST /drafts/<slug>/publish, POST /drafts/<slug>/discard, GET /api/genjobs/<id>. NO new always-on service (routes on the running KB app; serialized worker like the OCR ingest).
Claude on the box: natively installed at /home/levander/.local/bin/claude, logged in as levander (subscription). --model opusexplicit (never Sonnet); empty/error output → job fails loudly, nothing published. Shares the box with Frigate/OCR/vector → serialize, one at a time.
Proven live — two real articles:engine-cooling PUBLISHED at https://knowledgebase.taild4189d.ts.net/consolidated/engine-cooling/ (3 real source tags: chevy-tracker/geo-tracker-repair, suzuki-vitara/5door-supplement, suzuki-vitara/workshop-1988-1998; 49 images rendering; AI header present); parking-brake DRAFT (40 real citations, specs verbatim-traceable e.g. 20 to 25 kg (44 to 55 lbs) / 7 to 9 notches, nothing invented, honestly noted a source that contributed nothing + flagged a both-ways disagreement with ⚠). ⚠️ The parking-brake draft is still PENDING REVIEW — a normal review artifact the user may publish (Közzététel) or discard (Elvetés) from /drafts/<slug>; not lost work. Read-only held (source manuals + Qdrant untouched; only docs/consolidated/ added).
Key decisions: cite-everything/never-invent (every spec/number tagged [manual_id], verbatim numbers/units, conflicts both-ways + ⚠); on-demand one-at-a-time; draft→human-review→publish (never auto); opus; speccheck advisory (never a publish gate) ON TOP OF human review; AI-provenance banner + separate consolidated/ folder so an article is never mistaken for a manual.
https://home.taild4189d.ts.net — a landing dashboard (Hungarian “Elérhető rendszerek”, tiles Kamerák/Tudásbázis/Top képek/Router) on its own tag:telep tailnet node. The knowledgebase also became its own tag:telep node. Same reusable pattern as before: a dedicated second tailscaled per identity (--tun=userspace-networking, own --socket/--statedir) — see the starred tailnet-node gotcha and 2026-07-24-knowledgebase.
Windows drive + filebrowser (UPDATED 2026-07-21)
Windows NTFS partition mounted read-only at /mnt/win via ntfs3. RO deliberate (live Windows w/ fast-startup → RW would corrupt). ⚠️ Manual mount, not in fstab → gone on reboot.
⚠️ The device path is NOT /dev/nvme1n1p2 any more — NVMe enumeration flipped; on 2026-07-21 it was /dev/nvme0n1p2. lsblk -f and pick the 1.8T ntfs partition every time. An fstab/systemd entry must use UUID=, not a device node.
sudo modprobe ntfs3sudo mkdir -p /mnt/winsudo mount -t ntfs3 -o ro /dev/nvme0n1p2 /mnt/win # path per lsblkdocker restart filebrowser
→ options ro,relatime,uid=0,gid=0,iocharset=utf8.
⚠️ filebrowser must be restarted AFTER the mount. The container binds /mnt/win → /srv RO; if it started while /mnt/win was an empty unmounted dir it holds the stale pre-mount view and /srv looks empty inside the container while the host mount is fine. docker restart filebrowser. Every reboot = mount, then restart, in that order.
Proper fix: a systemd mount unit on UUID=, Before=docker.service — kills both the reboot gap and the restart dance.
filebrowser (docker, restart always) serves /mnt/win RO, no-login, at https://telep-mainframe.taild4189d.ts.net:8445 (ts serve :8445 → 127.0.0.1:8334). Chosen over Filestash (which has no local-FS backend). See filestash.
intruder-alarm.service / ~/intruder-alarm.py (python3 stdlib). Arms when no trusted phone on WiFi; escalates person/car to Telegram + TV siren.
Presence = union of router iwinfo assoclist (APs phy0-ap0+phy1-ap0 = ESSID telep1) andarp-scan on enp5s0. KEY: sleeping iOS phones stay WiFi-associated but stop answering ARP → arp-scan alone false-arms; the assoc table sees them. Failed scan is skipped (never “everyone left”).
⚠️ Camera-VLAN gotcha: AP phy1-ap1 = telep-cc (cameras) — excluded from presence, camera MAC hard-excluded, else the camera enrolls as a “phone” and the alarm never arms.
Enrollment: confirm-each via Telegram inline buttons. Names via mDNS (avahi-resolve) → dnsmasq reverse-DNS (dig -x @192.168.1.1) → eszköz-<last4>. Arms after all trusted absent 10min; disarms instantly. Intruder (armed): 🚨 BETÖRŐ Telegram + snapshot, 1×day/2×night. TV siren (always, armed or not) on every person/car:aplay -D plughw:1,3 ~/alarm/siren.wav (LG TV = ALSA card1 dev3). Commands /status /rename /untrust /arm /disarm /auto. State in ~/alarm/*.json.
Bandwidth throttle (NEW)
Device 192.168.1.135 (“F_M”, Android) was pulling ~46 Mbps. Capped to 15 Mbit/s via tc HTB on router br-lan, persistent in /etc/hotplug.d/iface/99-throttle-fm. tc-full+kmod-sched installed on the router. Remove: rm the hotplug script + tc qdisc del dev br-lan root. bmon (already installed on router) shows live per-interface rates.
Agent containers (Incus) — unchanged
Golden agent-base-img; agentctl (new/ls/rm/free/lock/claude/…); alpiq(rust,locked)/facekom(locked)/crypto(free)/*-test. nftables agentfw isolation, tag:dev-vm ACL. Control-plane per box (claude remote-control). Managed-deny on locked boxes.
exit-vpn container — ✅ WORKING (2026-07-22, was “STALLED”)
FaceKom “VirtualOffice” OpenVPN is up. The CA was never the problem — three real bugs: wrong server IP (.185 → 92.119.122.32), wrong crypto (AES-256-CBC / auth SHA1, not GCM/SHA256), and a corrupted auth file (the Hungarian jelszó:<TAB> label was being sent as part of the password → AUTH_FAILED).
Config /etc/openvpn/tt/farm.conf: proto tcp-client, remote 92.119.122.32 1194, data-ciphers AES-256-CBC:AES-256-GCM, auth SHA1, auth-user-pass auth-clean.txt (2 bare lines). Backups farm.conf.bak / auth-clean.txt.bak.
Full tunnel: server pushes no default route → added redirect-gateway def1 (installs 0.0.0.0/1+128.0.0.0/1 via tun0 plus a 92.119.122.32/32 host route via the real gateway). Office egress NATs to 92.119.122.32.
Persistent:openvpn.service is a /bin/true meta-unit and /etc/openvpn/tt/ isn’t scanned → ln -sf /etc/openvpn/tt/farm.conf /etc/openvpn/client/farm.conf; systemctl enable --now openvpn-client@farm.
Tailscale exit node: ts 1.98.9 in the container, node exit-vpn / 100.98.208.98, --advertise-exit-node --advertise-routes=192.168.105.0/24 --accept-dns=false, ip_forward via /etc/sysctl.d/99-tailscale.conf. Exit node approved in the admin console; the 192.168.105.0/24 subnet route is a separate checkbox and may still need ticking. Off by default — toggle with tailscale set --exit-node=exit-vpn / tailscale set --exit-node=.
tailscaled goes offline when the full tunnel rewrites the default route → fixed by tailscale-vpn-refresh.service (oneshot, PartOf/WantedBy=openvpn-client@farm.service, sleep 10; systemctl restart tailscaled).
FaceKom “VPN-only” hosts are NOT a DNS problem.npm.facekom.net resolves to the same public IP 92.119.122.189 from office DNS and 1.1.1.1; it sits behind an nginx source-IP ACL — 403 from the home WAN, 200 when sourced from the VPN. Routing, not name resolution.
Bundle + balazs’s screenshots live on the Mac at ~/telep/facekom-vpn/ (700/600). Both credential tars deleted from ~/Downloads. No secrets in this vault.
Repos / backup (NEW)
github.com/wowjeeez/telep-infra (PRIVATE) — infra snapshot: bin/ (camwall/pane/top4/agentctl scripts), services/ (intruder-alarm.py, top4-web.py), systemd/ units, nvr/ (sanitized configs). Secrets scrubbed (placeholders); rclone/keys/runtime-state git-ignored. Point-in-time — not live-synced; re-run the snapshot to refresh.
Obsidian vault auto-commits+pushes to github.com/wowjeeez/obsidian (the documenter does this).
OPEN / next
Power hardware (carry forward — all physical/BIOS access): BIOS Restore on AC Power Loss → Power On is the highest-leverage fix — auto-recovers the box from the daily ~06:00 mains blip like the router already does, removing the need for WOL / a physical press. Then a UPS + NUT for mains sags/graceful shutdown, and a bigger/better PSU for the load resets. Remote-wake (WOL) is NOT possible as configured (see the “Since last handover” block).
KB — full system now COMPLETE (phases 1–3 all DONE): phase 1 vectorize (2026-07-24-kb-vectorize-complete), phase 2 cluster+explorer (2026-07-24-kb-semantics-phase2-complete), phase 3 consolidated claude -p generation (2026-07-25-kb-consolidated-gen-phase3-complete) — DONE + proven live. Pipeline is OCR ingest → vectorize → cluster → on-demand cited consolidation. Still-open extras: the parking-brake draft is PENDING REVIEW (publish or discard from /drafts/<slug>); 27 image-only kick-fix PDFs still need an OCR pass; optional bridge = a claude -p relabel behind make_label for nicer cluster titles; deferred speccheck normalization minors (strip thousands-commas, ignore digits inside [citation] brackets); KB Hungarian/English toggle — MID-BRAINSTORM, PAUSED (Approach A chosen: mkdocs-static-i18n + Material language selector for the static site + a shared lang cookie for the Flask pages; HU/EN over UI chrome + cluster labels + generated consolidated articles; source manuals stay English. No spec doc written yet — resume via the brainstorming → writing-plans flow; see the “Since last handover (2026-07-28)” block); diagram understanding (wiring-diagram → structured/Mermaid via hybrid CV path-tracing + a vision model, accuracy-guarded, keep the original scan as source of truth) = a later phase.
Multi-pane control-remote (handover feature #2): a web page with pane buttons calling pane-switch, on its own ts-serve port. Not built. (Live screen-share was explored → HDMI-capture pane built → scrapped/reverted; panes are cctv+web only now.)
Persist /mnt/win via a systemd mount unit keyed on UUID= and ordered Before=docker.service — currently every reboot needs a manual mount plusdocker restart filebrowser.
filebrowser first-run / auth is no-login (tailnet-only) — fine as-is.
SECURITY — rotate these (values redacted here; they leaked earlier in transcripts + the obsidian repo history)
Tailscale API key (tskey-api-…) — revoke in admin console.
Telegram bot token — regenerate via BotFather if desired (used by frigate-notify + the alarm).
Frigate admin password (first-boot autogen) — change in UI.
Google Drive rclone OAuth — revoke at myaccount.google.com if desired.
Camera RTSP creds + telep-cc WiFi PSK also leaked.
NEW surface: the router LAN dropbear + ~/.ssh/router_alarm key (key-only, LAN-only — low risk, but note it exists).
⚠️ SESSION-HANDOVER.md and the vault are in the pushed wowjeeez/obsidian repo; keep literal secrets OUT of these notes going forward.
Key gotchas (hard-won)
⭐ Advertising IPv6 to a LAN whose upstream has NO working v6 makes macOS “slow” (not the bandwidth). macOS prefers IPv6 (RFC 6724 / Happy Eyeballs) and stalls 2–7 s per dual-stack site on the dead v6 path before IPv4 fallback. Diagnostic: curl -w shows time_namelookup instant but time_connect 2–7 s; curl -4 instant, curl -6/ping6 dead. Fix on the router: disable dhcp.lan.ra/dhcpv6/ndp + network.wan6. Fixed-wireless + double-NAT almost never carries working IPv6 — don’t advertise it. See 2026-07-28-ipv6-slow-internet.
⭐ Frigate birdseye is UNRELIABLE as a fixed multi-cam wall source once you have >2 cameras — it flip-flops between showing 2 and 4 cameras (drops panes by activity/state, by design). For a stable wall, composite the go2rtc substreams directly in the player. In mpv, an rtsp URL inside a movie= lavfi source needs colons escaped (rtsp\://...\:8554/...) and can’t take rtsp_transport (benign 461 Unsupported transport → UDP→TCP fallback); the mpv watchdog only covers the primary input, so a movie= pane can freeze if go2rtc restarts. Keep --hwdec=no. See 2026-07-28-camwall-4-substream-composite.
⭐ frigate-notify uses koanf, which SILENTLY IGNORES misplaced/unknown config keys — “Config validated! / App ready!” does NOT prove a filter works. Per-camera alert exclusion must be frigate.cameras.exclude (nested under frigate:, by camera id), NOT alerts.cameras.block / not top-level cameras:. A top-level attempt logged “validated” and still sent the alert. Always verify notification suppression against a real event (Event dropped - Camera Excluded), never config validation. See 2026-07-28-frigate-notify-camera-exclude.
⭐ On OpenWrt/busybox use setsid, not nohup (absent), to detach a self-reverting service swap. Keep a persisted binary backup on /overlay (survives reboot) and auto-revert if the new service isn’t online within N seconds — so a bad swap can’t lock you out of your only remote foothold. Used to upgrade the router’s tailscale 1.80.3→1.98.9 out-of-band; opkg still records 1.80.3-r1, so a future opkg upgrade could clobber the new binary (re-apply the static build or hold the package; backup at /overlay/ts-backup/). See the “Since last handover (2026-07-24)” block.
⭐ markdownify SILENTLY DROPS <img> inside <table> cells. Old sites use tables for LAYOUT, so nearly all figures get copied to disk but never referenced (11 refs for 122 pages initially). FIX: before markdownify, .unwrap() the layout containers (table/tbody/tr/td/th/center/font) so text+images flow linearly in doc order, rewrite each <img src> to the copied basename in place, wrap imgs in their own <p> (→ 528 refs). Keep a trailing ## Figures section only as a safety net. See 2026-07-24-kickfix-ingest.
⭐ “See photo here” TEXT links (<a href="X.jpg">) are separate from inline <img> and were 365/664 broken — their targets were never copied. FIX: also walk <a href>; if the href (URL-decoded, ../ resolved cross-topic) is an existing image, copy it under basename and rewrite href to basename (keep as a text link, don’t inline); leave external http(s)/mailto untouched; drop dead archive-local hrefs (→ 365→0 broken, images 449→741). See 2026-07-24-kickfix-ingest.
⭐ TWO separate power problems — don’t conflate them, and use the router-uptime discriminator. (a) Overnight MAINS outages: wall power drops, the router dies WITH the PC → fix is a UPS. (b) PSU-can’t-sustain-peak-load resets: wall power fine, PSU browns out under full load, box hard-resets while the router stays up 14h+ → fix is serialize jobs / cap threads / bigger PSU; a UPS does NOTHING for (b). Both show the same dead-stop journal signature — cross-check the router’s uptime to tell them apart. The default 16-thread CPU embedding maxing all cores reset the box in 2–6 min every time; OMP_NUM_THREADS=4/torch.set_num_threads(4) + nice + one heavy job at a time kept it up 45+ min and finished the index. See 2026-07-24-psu-load-resets.
⭐ HDBSCAN’s min_samples defaults to min_cluster_size — leaving it default made ALL section vectors noise (0 clusters). With min_cluster_size=2 and min_samples unset, HDBSCAN uses min_samples=min_cluster_size=2 and routes everything to the noise bucket. Set min_samples=1 explicitly for min_cluster_size=2 to actually form clusters. Looks like “the data has no structure” but is purely the default-coupling of the two knobs. See 2026-07-24-kb-semantics-phase2-complete.
⭐ Cross-source “span” is inflated when your id is <folder>/<manual> and one archive is split into many subfolders. The kick-fix archive is ONE source split into ~25 topic-subfolders, so ranking clusters by distinct manual_id made it look like ~25 manuals all covering a topic. Fix: define span by distinct TOP-LEVEL source folder (source_of = manual_id.split("/",1)[0]) → kick-fix collapses to 1 source while genuinely-separate manuals (suzuki-vitara/workshop vs /5door-supplement) still count. Keep source_manuals for display, rank by (span=distinct sources, #manuals, size). Also add a boilerplate label stoplist (is_boilerplate()) or cover/title pages (“SUZUKI”) and generic headings (“GENERAL DESCRIPTION”/“DIAGNOSIS”) sort to the top. And 51% noise singletons is EXPECTED/correct for a dedup goal — sections unique to one manual aren’t duplicated knowledge. See 2026-07-24-kb-semantics-phase2-complete.
⭐ Consolidated repair-article generation: image basename collision across manuals silently swaps in the WRONG diagram. Every marker-OCR’d manual names its images the same way (_page_62_Figure_16.jpeg, etc.), so a consolidated article pulling images from MULTIPLE manuals would copy the wrong manual’s figure — a real accuracy failure for a repair doc. Fix: before generation, rewrite each source’s image refs to SOURCE-UNIQUE names (safe_slug(manual_id) + "__" + basename) and build an image_map (unique-name → real file) so copies resolve 1:1 to the correct source image (verified 4 refs → 4 correct prefixed copies). See 2026-07-25-kb-consolidated-gen-phase3-complete.
⭐ A number-token spec-check’s “false flags” are mostly REAL verbatim-deviations — keep it ADVISORY, never a publish gate. speccheck flags any draft number not string-present in the sources; these fire mainly when the model reformats a number (1,000→1000, 40→40.0), which the never-invent prompt forbids, so a “false flag” is usually a real deviation worth surfacing. It also harmlessly flags year-digits from citation slugs (1988/1998 from [...workshop-1988-1998]). Human review is the backstop, not the checker. Also: start_worker must be idempotent (module flag) or a second call spawns a concurrent worker and breaks one-at-a-time serialization; safe_slug guarantees a non-empty single path component so a degenerate label can’t make publish()’s rmtree wipe the whole consolidated/ folder. claude -p on the box: native installer → ~/.local/bin, claude login (subscription), then headless claude -p --model opus on stdin; --model opus explicit + fail-loud on empty/error. See 2026-07-25-kb-consolidated-gen-phase3-complete.
The KB semantics explorer is a READ-ONLY route on the existing knowledgebase app, not a new service — the whole clustering pipeline only scrolls points OUT of Qdrant and writes a local clusters.json (never upsert/delete/create), and the label layer is pluggable behind make_label (future claude -p relabel). Deliberately no new always-on service on the power-flaky box. See 2026-07-24-kb-semantics-phase2-complete.
Always md5-dedup PDF downloads BEFORE (GPU-serialised) OCR — many were exact dupes or same-content-different-name (workshop ×2, 99501 ×2, parts-catalogue ×2); zips sometimes contain the PDF. See 2026-07-24-knowledgebase.
⭐ Debian 13’s stock initramfs (initramfs-tools) does NOT consume systemd-cryptenroll TPM2 tokens.systemd-cryptenroll --tpm2-device=… enrolls a valid TPM keyslot, but update-initramfs warns ignoring unknown option 'tpm2-device' and the box still prompts at boot — Debian’s classic cryptsetup hook can’t read a systemd-tpm2 token. For TPM LUKS auto-unlock on Debian use clevis (clevis-initramfs + clevis-tpm2, clevis luks bind … tpm2 '{}'), NOT systemd-cryptenroll + crypttab tpm2-device=. See 2026-07-24-luks-tpm-autounlock.
⭐ Adding ANY new LUKS keyslot needs the EXISTING passphrase — an agent cannot do it for the user. Both clevis luks bind and systemd-cryptenroll authorize the new slot against a current keyslot, so someone who knows the passphrase must type it. And the op is genuinely slow (argon2 KDF keyslot-add + TPM seal = tens of seconds) — an impatient Ctrl-C leaves a half-run process (which can orphan and hold the header lock). Let it finish. See 2026-07-24-luks-tpm-autounlock.
A stuck systemd-crypt* process (SSH killed mid-prompt) survives holding the LUKS header lock AND /dev/tpmrm0 and keeps re-broadcasting a systemd-ask-password request — silently blocking any later clevis luks bind (header locked). Fix: SIGKILL the orphan (SIGTERM may not clear one mid-TPM-op), confirm fuser /dev/nvme0n1p3 + lsof /dev/tpmrm0 clear and luksDump still shows the passphrase slot, before retrying. See 2026-07-24-luks-tpm-autounlock.
⭐ A separate tailnet identity on one host = a second tailscaled with --tun=userspace-networking, its own --socket and --statedir. Then tailscale --socket=… up --hostname=<name> (one interactive auth) + tailscale --socket=… serve …. Gives <name>.<tailnet>.ts.net as its own device without a container or GPU passthrough — the app + OCR stay on the host where the GPU + marker venv are. Used for the knowledgebase node — 2026-07-24-knowledgebase.
⭐ tailscale serve --https fails with “no TailscaleVarRoot” when the second tailscaled was started with only --state=<file>. Symptom is a TLS handshake error — curl gets 000 even with -k while TCP/ping succeed, so it looks like a listener/firewall issue but isn’t. Cause: with only a state file tailscaled has no var root to store the Let’s Encrypt cert. Fix: start it with --statedir=<dir> instead (the state file lives inside the statedir, so the node stays authenticated); then tailscale --socket=… cert <name> provisions the cert and TLS works — 2026-07-24-knowledgebase.
A journal that stops dead with no Stopping/panic/thermal trace = power was physically removed, not a software or OS shutdown. Software shutdown always logs Stopping; a panic always logs a trace — their absence is the diagnosis. See 2026-07-23-mains-power-shutdowns.
To tell a PC PSU fault from a mains event, cross-check an independent always-on device’s boot time (e.g. the router). Derive it from date +%s minus /proc/uptime. Same-timestamp death = mains; only-the-PC died = its own PSU.
A desktop that “turns off and stays off” while other gear silently recovers is the BIOS Restore on AC Power Loss = stay off policy, not a different failure. Embedded gear (router) auto-powers-on; the PC needs the policy flipped to match.
NVDEC on the 3080 corrupts camwall live feeds (magenta) under GPU load → --hwdec=no. Grab :0 framebuffer to actually see the TV.
Frigate birdseye is NVENC-encoded (h264_nvenc, inherits global hwaccel) → the encoder session can wedge and corrupt the wall’s live panes blue/purple even at GPU idle; docker compose restart frigate clears it, blank hwaccel_args for a durable libx264 fix. Isolate with: grab birdseye + source cam + /api/<cam>/latest.jpg — all-clean-but-birdseye = the encode stage. See 2026-07-18-birdseye-nvenc-blue-glare.
Never letterbox in the wall’s lavfi --vf (pad config-fail kills the composite) → letterbox canvas-side into 880×495 instead.
X-Forwarded-For is the LAST quoted field before request_time= in an nginx access line, not the first. Leftmost-match (re.search(r'"([^"]*)" request_time=') is spoofable: a client injecting a literal " into its User-Agent/Referer plants an earlier "…" request_time= occurrence and forges the captured viewer IP. Default nginx escapes quotes so it’s safe in practice, but take findall(...)[-1] (the field truly adjacent to request_time=) so you don’t depend on the log-escape setting. Applies to ANY nginx-log parsing that trusts a quoted field — 2026-07-23-frigate-viewer-alert.
Never spawn tailscale status (or any ~10s subprocess) per log line. Refreshing a name cache on every unknown-IP line becomes one blocking subprocess PER line from an active viewer when tailscale status is momentarily failing or the IP isn’t listed — a self-DoS. Rate-limit miss-triggered refreshes (≤ once/60s) on top of the periodic refresh — 2026-07-23-frigate-viewer-alert.
mpv destroys osd-overlay entries the moment the IPC client disconnects — connect→send→close silently does nothing (mpv still replies {"error":"success"}). The IPC connection must be persistent, and you must drain mpv’s event stream or the socket buffer fills. Reconnect-on-failure is what makes the overlay self-heal after a camwall restart.
camwall-reload is unusable for live data — its loadfile restarts video playback (fine weekly for TOP KÉPEK, would stutter the live cameras every 5s). Use osd-overlay over /run/camwall.sock: touches neither the stream nor the filter chain. Note /run/camwall.sock is srw------- root root → overlay pushers must run as root.
iwinfo assoclist has no byte counters; iw dev <ap> station dump does (rx/tx bytes + signal, iw already installed) → per-device bandwidth needs no nftables/conntrack/nlbwmon.
Station byte counters are from the AP’s point of view: tx bytes = the device’s DOWNLOAD, rx bytes = its upload. The WAN interface is NOT inverted. Easiest bug to ship.
SSH ControlMaster masks credential loss — deleting the router key changes nothing while the multiplexed ControlPersist connection lives (the key file is never re-read). To test degradation you must also drop the control socket (/run/wifi-usage/router.sock).
Sleeping iOS phones drop ARP but stay WiFi-associated → presence needs the router assoc table, not just arp-scan.
Camera VLAN AP (phy1-ap1/telep-cc) leaks the camera into any assoc-based device list → exclude it.
Telegram sends take ~15s on the flaky WAN → answer callbacks FIRST, never hold a lock during network I/O.
dig/subprocess error text can leak into data if stdout isn’t validated → strict hostname regex + stderr to devnull.
iOS AirPrint can’t work over Tailscale (mDNS/L3). Mac-by-IP works.
macOS: bare ssh broken (use /usr/bin/ssh); iCloud Private Relay breaks tailnet MagicDNS in iOS Safari.
NVMe device nodes on the mainframe flip across reboots (nvme0n1 ↔ nvme1n1) → lsblk -f before mounting, UUID= in fstab.
A docker bind-mount captures the host dir at container start — mounting a filesystem under it afterwards leaves the container with an empty stale view. Mount first, then (re)start the container.
Two OpenVPN servers live on 92.119.122.0/24 — connecting to the wrong one (.185 vs .32) presents a different CA and looks exactly like a “bad CA bundle”. Verify the gateway IP before blaming certs.
Vendor auth.txt is labelled + tab-delimited (user: X, jelszó:<TAB>pw) → the auth-user-pass file must be exactly 2 bare lines or OpenVPN sends the label as part of the password (AUTH_FAILED).
openvpn-client@.service is hardened with a process limit → any up/route-up script fails with unable to fork … (errno=11) and kills the tunnel; use a separate systemd unit instead.
Tailscale exit nodes don’t carry RFC1918 — private LAN access needs an approved subnet route, not just the exit node.
Full-tunnel route changes strand tailscaled’s control connection → node shows offline; restart tailscaled after the tunnel comes up.
When testing a route change, curl may resolve a different (anycast) IP than the one you routed → always pin with curl --resolve host:443:<IP>.
TensorRT dead on amd64 (0.17)→ONNX; Tapo truncates >14-char pw, Basic auth only, reboot after cred change; /cam1+/cam2 RTSP = same lens (use /stream1 vs /stream6); Docker FORWARD=DROP breaks incus → DOCKER-USER fix (agent-net-fix.sh).