Box is UP — booted 2026-07-24 13:55 after someone physically pressed power following a morning mains outage. clevis TPM auto-unlock worked again (unattended encrypted-root unlock). All services active: knowledgebase, home-portal, wifi-usage, intruder-alarm, camwall, tailscaled-knowledgebase, tailscaled-home; kb-qdrant + frigate containers Up (frigate.service shows inactive because Frigate is docker, not systemd — expected). Qdrant manuals collection: 2839 points. A resume OCR/vectorize job is in-flight (see “In-flight” below) — check it finished.
Since last handover (2026-07-24)
Delta since the last handover commit a197813.
Router Tailscale upgraded 1.80.3 → 1.98.9
telep-router tailscale hand-upgraded to a static arm64 1.98.9 build (the opkg feed only carries 1.80.3). Router online, 8 peers, verified over SSH served by the new binary.
Old binaries backed up at /overlay/ts-backup/ (persists across reboot). Done via a detached self-reverting swap script that rolls back to the backup if the new binary isn’t online within 45s.
setsid, not nohup, for a self-reverting service swap on OpenWrt/busybox
busybox has no nohup — use setsid to detach a swap script that could otherwise kill your only remote foothold. Keep a persisted binary backup on /overlay (survives reboot) and auto-revert on failure (roll back if the new service isn’t online within N seconds), so a bad swap can’t lock you out of the router.
The router tailscale binary is out-of-band — opkg may clobber it
The 1.98.9 binary was hand-installed; opkg still RECORDS 1.80.3-r1. A future opkg upgrade would silently restore 1.80.3 over the new binary. Re-apply the static build (or hold the package) after any opkg upgrade. Backup lives at /overlay/ts-backup/.
Daily ~06:00 mains die-off (NEW observation — a mains signature, not PSU)
The box appears to lose power around 06:00 most mornings — deaths clustered Jul 19 ~06:59, Jul 23 ~06:36, Jul 24 morning. These take the router down too (router uptime resets with the box) → MAINS-level, distinct from the PSU-load resets (where the router survives 14h+).
Suspected: a scheduled utility event — a Hungarian off-peak/controlled (vezérelt/GEO) tariff relay switching a controlled circuit ~06:00, OR a timed high-draw appliance (electric bojler/heater) on the same circuit browning out the feed. NOT confirmed.
To confirm next time the box is up: journalctl --list-boots + correlate every death with router reboots for clock-regularity.
Daily ~06:00 mains die-off is UNCONFIRMED
Three mornings of ~06:00 power loss that take the router down too (= mains, not PSU). Cause not yet proven — suspect a scheduled utility relay (vezérelt/GEO tariff) or a timed high-draw appliance on the shared circuit. Confirm with journalctl --list-boots vs router reboot times for clock-regularity. SEPARATE from the PSU-under-load resets (the router SURVIVES those — see 2026-07-24-psu-load-resets).
Remote-wake (WOL) is NOT possible as configured
After a mains outage the box needs a physical power-button press. WOL was never enabled (BIOS + NIC); the router has no etherwake/wakeonlan; and after ~6h off the box’s MAC is unknown (DHCP lease expired, no static reservation, ARP cleared).
To make remote-wake possible later: (a) BIOS enable Wake-on-LAN, (b) persistent ethtool -s enp5s0 wol g, (c) a static DHCP reservation on the router so the MAC is always known, (d) a WOL tool on the router.
Better fix: BIOS Restore on AC Power Loss → Power On so the box auto-recovers from the daily mains blip like the router already does (needs one physical BIOS visit).
In-flight — check it finished next session
~/ocr/resume_extras.sh (setsid-detached; sentinels ~/ocr/resume_extras.done / .failed): finishes the interrupted extras OCR. body-measurements already OCR’d + seeded into suzuki-vitara; this OCRs the Sidekick parts-catalogue (sidekick_parts_89_98.pdf) → seeds suzuki-sidekick/parts-catalogue-89-98 → rebuilds the site → then vectorizes BOTH new manuals (body-measurements + parts-catalogue) into Qdrant (thread-capped OMP_NUM_THREADS=4 + nice, per the PSU lesson → will push manuals past 2839 points).
If it died (another power event), re-runbash ~/ocr/resume_extras.sh.
Access (do this first)
macOS gotcha: bare ssh is a broken shell wrapper. Always use /usr/bin/ssh.
Mainframe: /usr/bin/ssh levander@telep-mainframe (tailnet) or …@192.168.1.123 (LAN). Passwordless sudo.
Router: /usr/bin/ssh root@telep-router (OpenWrt, busybox — no timeout, classic ip). NEW: a LAN-only, key-only dropbear now also listens on 192.168.1.1:22 (added so the mainframe can read WiFi presence). GOTCHA: enabling it rotated the router’s SSH host key → the Mac’s known_hosts entry for telep-router is now stale (Host key verification failed); remove the offending line to reconnect.
Mainframe→router (for the alarm): /usr/bin/ssh -i /home/levander/.ssh/router_alarm root@192.168.1.1.
Agent containers from Mac: /usr/bin/ssh alpiq (also facekom, crypto, alpiq-test, facekom-test) via ProxyCommand through mainframe incus.
Secrets live on the Mac in ~/telep/ (all 600). NOT on the mainframe (the alarm reads the bot token from ~/nvr/frigate-notify/config.yml).
Double-NAT behind 192.168.254.1. WAN outbound is intermittently flaky (causes ~15s Telegram send latency + occasional dig/API failures — not the code, the uplink).
⚠️ Recurring “turns off overnight” = MAINS POWER loss, not the PC (2026-07-23). Confirmed mains: the telep-router (separate PSU) died at the same instant (mainframe journal dead 06:36:10, router booted 06:36:27). Journal stops dead with no Stopping/panic/thermal trace; temps cool; no poweroff timers. The PC stays dark only because BIOS Restore on AC Power Loss = stay-off (the router auto-powers-on, so its recovery is invisible). Fix: a ~600–900 VA UPS + NUT (logs sags, graceful shutdown); free stopgap = flip BIOS AC-loss to Power On (physical BIOS access only, not over SSH); also rule out an overloaded shared circuit (i9+3080+TV). ext4 clean so far but repeated hard cuts risk corruption. Full detail in 2026-07-23-mains-power-shutdowns.
⚠️ SECOND, distinct power problem (NEW 2026-07-24): PSU-can’t-sustain-peak-load resets. The box crash-rebooted ~3× in 30 min (04:11/04:20/04:32) while running MANY heavy jobs at once (GPU OCR + CPU embedding + kick-fix conversion + Frigate). NOT thermal (CPU 39°C/GPU 47°C, no MCE), NOT mains — the router stayed up 14h+ straight through (the router-uptime cross-check is the discriminator: mains = router dies too, PSU = only the PC resets). Dead-stop journal signature. Root cause: the PSU can’t hold peak combined i9-12900K + RTX 3080 draw; specifically the default 16-thread CPU embedding maxing all cores reset the box in 2–6 min every time. A UPS does NOT fix this — that’s for (a) mains. Fix (b): serialize heavy jobs + cap threads (OMP_NUM_THREADS=4/torch.set_num_threads(4) + nice → box stayed up 45+ min, completed the full index); real fix = a bigger/better PSU. Full detail in 2026-07-24-psu-load-resets.
⚠️ THIRD signature — a daily ~06:00 mains die-off (NEW 2026-07-24, UNCONFIRMED). The mains outages (a) cluster around 06:00 most mornings (Jul 19 ~06:59, Jul 23 ~06:36, Jul 24 morning) and take the router down too (= mains, not PSU). Suspected: a scheduled utility relay (Hungarian vezérelt/GEO off-peak tariff) or a timed high-draw appliance browning the shared circuit — not proven. Confirm via journalctl --list-boots vs router reboot regularity. Remote-wake is impossible as configured (WOL off, MAC unknown after lease expiry) — the real fix is BIOS Restore on AC Power Loss → Power On. Full delta in the “Since last handover (2026-07-24)” block above + 2026-07-23-mains-power-shutdowns.
Drives (2026-07-21): ⚠️ NVMe device nodes flip between reboots — always lsblk -f, never hardcode. Currently nvme1n1 = 3.6T OS disk (p1 /boot/efi, p2 /boot ext4, p3 LUKS+LVM); nvme0n1 = 1.8T Windows NTFS disk, C: on nvme0n1p2 (~886G used). sda 447G unused (single 16M part, no fs). sdb 115.5G = Hiren’s BootCD PE USB stick (UDF, HBCD_PE_x64).
Disk encryption / auto-unlock (NEW 2026-07-24 — full detail in 2026-07-24-luks-tpm-autounlock): root is LUKS2 on LVM (container /dev/nvme0n1p3, mapper nvme1n1p3_crypt, UUID a3a8e37d-79fa-484b-bc3f-40c56df95337). Was passphrase-only → sat at an unlock prompt after every reboot (unreachable over SSH, no dropbear-initramfs). Now has clevis TPM2 auto-unlock: keyslot 1 = TPM (bound with empty PCRs, sudo clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'), keyslot 0 = passphrase (fallback, untouched). TPM /dev/tpmrm0 (MSFT0101), Secure Boot OFF. Check:clevis luks list -d /dev/nvme0n1p3. Header backed up first at ~/telep/luks-header-nvme0n1p3-20260723.img (600, on the Mac). ✅ Reboot test PASSED IN PRODUCTION 2026-07-24 — the box auto-unlocked root through a 3-reboot PSU-load storm, no passphrase, all services self-recovered (see 2026-07-24-psu-load-resets). ⚠️ Auto-unlock only helps once the box boots — BIOS Restore on AC Power Loss is still stay-off, so a mains cut still needs a button press (see 2026-07-23-mains-power-shutdowns). Empty-PCR + Secure-Boot-off tradeoff: protects against bare-drive theft, NOT whole-box theft.
Frigate NVR (~/nvr/ on mainframe)
docker compose: frigate (0.17.2, ONNX yolov9-s @640 on GPU, NOT tensorrt) + frigate-notify (polls API 15s, pushes Telegram, no MQTT). Media /srv/frigate.
One dual-lens Tapo TC47 @ 192.168.30.119 on isolated telep-cc VLAN. /stream1+/stream2 fixed lens, /stream6+/stream7 PTZ. go2rtc restreams cam1/cam2 + birdseye. Tracks person+car, face_recognition on.
Telegram bot “Dezsi az őr” @tlphlyortbot → group -1004475187307. Alerts on person+car (no retry — WAN blip = lost alert).
Frigate viewer alert (NEW — full detail in 2026-07-23-frigate-viewer-alert):frigate-viewer-alert.service (runs as levander, NOT root; enabled, survives reboot) follows docker logs -f --since 0m frigate, reads the real client IP from each nginx access line’s X-Forwarded-For (set by tailscale serve), maps 100.x → tailscale device name, and Telegrams 👁 <device> (100.x) opened Frigate — HH:MM[ · live: <cam>] when someone opens the UI. Reuses the frigate-notify bot+chat (token/chatid read at runtime from ~/nvr/frigate-notify/config.yml, never copied) so viewer pings land in the same chat as person/car alerts. Internal automation has XFF - → excluded by construction. Per-source-IP 5-min session cooldown (an open tab polls constantly → edge-trigger). Code /home/levander/frigate-viewer-alert/ (python3 stdlib, 34 tests). Only unverified leg = a real browser session (can’t be faked into the live stream) — user 3-step manual test in the note.
camwall-mpv.sh now uses --hwdec=no (software decode). ⚠️ The RTX 3080’s NVDEC was corrupting the live feeds → whole wall went magenta/green; software decode on the 12900K fixed it (GPU contention w/ Frigate detection was the cause). Debug it by grabbing the X framebuffer: sudo DISPLAY=:0 XAUTHORITY=<xorg -auth> ffmpeg -f x11grab -i :0.0 ….
Pane system:pane-runner.sh reads /srv/frigate/pane (cctv|web), pane-switch <pane> [url] writes state + pkills current app. cctv → camwall-mpv.sh composite; web → chromium kiosk.
WiFi usage strip (NEW — full detail in 2026-07-22-wifi-usage-strip):wifi-usage.service (root, enabled, survives reboot) polls the router every 5s over one persistent SSH connection, diffs iw station dump byte counters into live Mbps, and pushes an ASS osd-overlay into the running mpv. Renders a strip at the bottom of the cctv pane (y 1880–2140, below TOP KÉPEK) with every telep1 device’s ↓/↑ plus a 5G WAN total header; 7 tiles, then “6 busiest + +N további”. Code /home/levander/wifi-usage/ (python3 stdlib, 44 tests). Names reused from the alarm’s ~/alarm/trusted.json → DHCP hostname → eszköz-<last4>. Router failure keeps last values with ⚠, then — after 60s — never blanks the wall.
Reload wall images without restart:camwall-reload (mpv IPC loadfile). ⚠️ It reloads through the current --vf — do NOT put a filter in the vf that can fail to configure (e.g. pad with a mismatched size), it kills the whole composite → raw birdseye only. Filter changes need systemctl restart camwall, not camwall-reload.
Top-képek picker (~/top4-web.py, service top4-web, ts serve :8443)
Slot-replace UI. NEW: in-browser CROP — arm a slot, click a detection → crop modal (free-form, all-8 handles, drag to move). The crop is letterboxed to exactly 880×495 in the browser canvas (black bars for non-16:9) and posted as a finished JPEG. This keeps the wall’s scale=880:495 filter simple/unbreakable (the pad-in-wall-filter approach crashed the wall — see camwall gotcha). New endpoints /full/<id>, /set-slot-image, /export.
NEW: “Exportálás Drive-ra” button → /export runs top4-export. top4-export now writes gdrive:top_képek/<date>_<HH-MM-SS>/ (timestamped, no overwrite). top4-export.timer Mon 08:00 unchanged.
Printer (NEW)
HP LaserJet M203dw @ 192.168.1.126 (NPIB8AE8C), driverless IPP, set as CUPS default on the mainframe. lp file.pdf just works locally.
Remote access: router advertises subnet route 192.168.1.126/32 (approved). Mac: add by IP (ipp://192.168.1.126:631/ipp/print, IPP-Everywhere). iPhone AirPrint over Tailscale is IMPOSSIBLE (mDNS doesn’t cross Tailscale; iOS print sheet is discovery-only) — use HP Smart / a direct-IP app instead.
What: tailnet-only manual library + OCR ingest portal. mkdocs-material static site (folder-tree browse, full-text search, dark mode, phone-responsive, lazy images) of OCR’d PDF manuals, plus a Flask /upload portal that OCRs a PDF and adds it to the library.
Its OWN tailnet node:https://knowledgebase.taild4189d.ts.net — NOT a port on the telep-mainframe node. Runs on the mainframe host (GPU + marker venv live there); only the tailnet identity is separate, via a dedicated second tailscaled.
Two systemd units (both enabled, survive reboot):knowledgebase.service (Flask+waitress app on 127.0.0.1:8092, User=levander not root) and tailscaled-knowledgebase.service (the dedicated node’s tailscaled, --tun=userspace-networking, own --socket/--statedir).
Ingest reuses the ~/ocr marker venv as a subprocess (marker_single, TORCH_DEVICE=cuda) — no duplicate torch/model install (see scanned-pdf-to-markdown-marker). Single background worker, GPU-serialised (shared with Frigate); job states queued→ocr→splitting→building→done/failed. mkdocs builds to a temp dir then os.rename-swaps site/ (a broken source never takes down the live site).
Add manuals: the /upload portal (pick folder + name; PDF magic-byte check, 500 MB cap) or seed_import.py (import already-OCR’d marker output without re-OCR). Library source manuals-src/docs/<folder>/<name>/ → mkdocs auto-nav.
Library now GREATLY expanded (2026-07-24):suzuki-vitara/ (5door-supplement, supplement-61a40/99501, workshop-1988-1998 835pg, wiring-diagrams/99512, body-measurements), chevy-tracker/ (owners-1995, geo-tracker-repair), suzuki-sidekick/ (wiring-1996, parts-catalogue-89-98). OCR pipeline: marker (~/ocr/venv, marker_single --force_ocr, TORCH_DEVICE=cuda) → seed_import.py → build. Manuals queued as CHAINED nohup scripts (each waits on the prior’s .done sentinel) to serialize the single GPU. ⚠️ md5-dedup downloads BEFORE OCR — many were exact dupes or same-content different-names (workshop ×2, 99501 ×2, parts-catalogue ×2); zips sometimes contain the PDF (sidekick wiring was a zip of wiring2.pdf). Full detail 2026-07-24-knowledgebase.
kick-fix.com archive ingest (NEW — full detail 2026-07-24-kickfix-ingest): 112 MB offline archive of the dead kick-fix.com Suzuki repair site → docs/kick-fix/ (25 topics, 127 articles, 741 images, 8 text-layer PDFs; 27 image-only PDFs deferred to OCR). Dead-site archive = no ToS/robots issue. Script /home/levander/knowledgebase/kickfix_ingest.py (KB venv + markdownify + beautifulsoup4 + pypdfium2). See gotchas.
Phase 1 DONE: every manual section + kick-fix article embedded into a local Qdrant vector DB. 2839 chunks across 33 manual/kick-fix collections; cross-source semantic search works and blends official manuals + community guides (e.g. “depressurize the fuel line” → kick-fix injectors 0.79 + vitara Engine Fuel).
Code/home/levander/kb-vectors/ (own venv, runs as levander): chunker.py, embed.py, index.py, search.py, kbvec.py (CLI: index/search/stats). Embeddings: local BAAI/bge-large-en-v1.5, CPU torch, 1024-dim cosine, normalized; query prefix "Represent this sentence for searching relevant passages: ". Store: Qdrant container kb-qdrant on 127.0.0.1:6333/:6334, storage ~/kb-vectors/qdrant-storage, collection manuals. qdrant-client 1.18. Idempotent per-manual (delete-by-manual-filter + deterministic uuid5 ids).
⚠️ The 16-thread CPU embedding is what reset the box (see PSU-load bullet above) — run capped to 4 threads + serialized. Suzuki-forums crawler DROPPED (Tollbit-gated against AI crawlers).
Phase 2 DONE: fully-local semantic SECTION clustering over the phase-1 vectors + a read-only web explorer, NO LLM. Pipeline: scroll all 3252 chunk vectors from Qdrant manuals (READ-ONLY) → group by (manual_id,page) → L2-normalized mean-pool → ~1248 SECTION vectors → HDBSCAN(min_cluster_size=2, min_samples=1, euclidean on unit-norm ≈ cosine) → representative-heading labels → clusters.json. Result: 194 clusters (post boilerplate filter), 24 cross-SOURCE (span≥2), 631 noise singletons (noise is EXPECTED/correct for a dedup goal). Top clusters are real cross-make overlaps (WINDOW REGULATOR spanning chevy-tracker + suzuki-vitara/5door + workshop; ENGINE COOLING, PARKING BRAKE, THROTTLE BODY).
Code/home/levander/kb-vectors/ (reuses the phase-1 venv, runs as levander): new sectionvecs.py, labels.py, cluster.py, kbclust.py (+ tests). CLI:kbclust.py build/list/show <id>; build regenerates clusters.json on demand (thread-capped per the PSU lesson) after the corpus changes.
Explorer:GET /semantics + GET /api/clusters folded into the EXISTING knowledgebase Flask app (/home/levander/knowledgebase/app.py) — NO new always-on service on the power-flaky box. Live at https://knowledgebase.taild4189d.ts.net/semantics (Hungarian “Szemantika”); each cluster shows label + N források + members linking into the KB pages; cross-source first; empty state if clusters.json missing. Dashboard tile 🧭 Szemantika on home.taild4189d.ts.net.
Key decisions: section-level unit (not chunk); local heading labels pluggable behind make_label (a claude -p relabel drops in later); ranked by cross-SOURCE span so dedup targets surface first.
⚠️ Phase 3 (claude -p consolidated generation) still NOT started — clusters.json is its retrieval bundle (each cross-source cluster = the sources to hand claude for a topic). Needs Claude Code installed + logged-in (subscription) on the box — still not done. Optional bridge = a claude -p relabel pass behind make_label.
https://home.taild4189d.ts.net — a landing dashboard (Hungarian “Elérhető rendszerek”, tiles Kamerák/Tudásbázis/Top képek/Router) on its own tag:telep tailnet node. The knowledgebase also became its own tag:telep node. Same reusable pattern as before: a dedicated second tailscaled per identity (--tun=userspace-networking, own --socket/--statedir) — see the starred tailnet-node gotcha and 2026-07-24-knowledgebase.
Windows drive + filebrowser (UPDATED 2026-07-21)
Windows NTFS partition mounted read-only at /mnt/win via ntfs3. RO deliberate (live Windows w/ fast-startup → RW would corrupt). ⚠️ Manual mount, not in fstab → gone on reboot.
⚠️ The device path is NOT /dev/nvme1n1p2 any more — NVMe enumeration flipped; on 2026-07-21 it was /dev/nvme0n1p2. lsblk -f and pick the 1.8T ntfs partition every time. An fstab/systemd entry must use UUID=, not a device node.
sudo modprobe ntfs3sudo mkdir -p /mnt/winsudo mount -t ntfs3 -o ro /dev/nvme0n1p2 /mnt/win # path per lsblkdocker restart filebrowser
→ options ro,relatime,uid=0,gid=0,iocharset=utf8.
⚠️ filebrowser must be restarted AFTER the mount. The container binds /mnt/win → /srv RO; if it started while /mnt/win was an empty unmounted dir it holds the stale pre-mount view and /srv looks empty inside the container while the host mount is fine. docker restart filebrowser. Every reboot = mount, then restart, in that order.
Proper fix: a systemd mount unit on UUID=, Before=docker.service — kills both the reboot gap and the restart dance.
filebrowser (docker, restart always) serves /mnt/win RO, no-login, at https://telep-mainframe.taild4189d.ts.net:8445 (ts serve :8445 → 127.0.0.1:8334). Chosen over Filestash (which has no local-FS backend). See filestash.
intruder-alarm.service / ~/intruder-alarm.py (python3 stdlib). Arms when no trusted phone on WiFi; escalates person/car to Telegram + TV siren.
Presence = union of router iwinfo assoclist (APs phy0-ap0+phy1-ap0 = ESSID telep1) andarp-scan on enp5s0. KEY: sleeping iOS phones stay WiFi-associated but stop answering ARP → arp-scan alone false-arms; the assoc table sees them. Failed scan is skipped (never “everyone left”).
⚠️ Camera-VLAN gotcha: AP phy1-ap1 = telep-cc (cameras) — excluded from presence, camera MAC hard-excluded, else the camera enrolls as a “phone” and the alarm never arms.
Enrollment: confirm-each via Telegram inline buttons. Names via mDNS (avahi-resolve) → dnsmasq reverse-DNS (dig -x @192.168.1.1) → eszköz-<last4>. Arms after all trusted absent 10min; disarms instantly. Intruder (armed): 🚨 BETÖRŐ Telegram + snapshot, 1×day/2×night. TV siren (always, armed or not) on every person/car:aplay -D plughw:1,3 ~/alarm/siren.wav (LG TV = ALSA card1 dev3). Commands /status /rename /untrust /arm /disarm /auto. State in ~/alarm/*.json.
Bandwidth throttle (NEW)
Device 192.168.1.135 (“F_M”, Android) was pulling ~46 Mbps. Capped to 15 Mbit/s via tc HTB on router br-lan, persistent in /etc/hotplug.d/iface/99-throttle-fm. tc-full+kmod-sched installed on the router. Remove: rm the hotplug script + tc qdisc del dev br-lan root. bmon (already installed on router) shows live per-interface rates.
Agent containers (Incus) — unchanged
Golden agent-base-img; agentctl (new/ls/rm/free/lock/claude/…); alpiq(rust,locked)/facekom(locked)/crypto(free)/*-test. nftables agentfw isolation, tag:dev-vm ACL. Control-plane per box (claude remote-control). Managed-deny on locked boxes.
exit-vpn container — ✅ WORKING (2026-07-22, was “STALLED”)
FaceKom “VirtualOffice” OpenVPN is up. The CA was never the problem — three real bugs: wrong server IP (.185 → 92.119.122.32), wrong crypto (AES-256-CBC / auth SHA1, not GCM/SHA256), and a corrupted auth file (the Hungarian jelszó:<TAB> label was being sent as part of the password → AUTH_FAILED).
Config /etc/openvpn/tt/farm.conf: proto tcp-client, remote 92.119.122.32 1194, data-ciphers AES-256-CBC:AES-256-GCM, auth SHA1, auth-user-pass auth-clean.txt (2 bare lines). Backups farm.conf.bak / auth-clean.txt.bak.
Full tunnel: server pushes no default route → added redirect-gateway def1 (installs 0.0.0.0/1+128.0.0.0/1 via tun0 plus a 92.119.122.32/32 host route via the real gateway). Office egress NATs to 92.119.122.32.
Persistent:openvpn.service is a /bin/true meta-unit and /etc/openvpn/tt/ isn’t scanned → ln -sf /etc/openvpn/tt/farm.conf /etc/openvpn/client/farm.conf; systemctl enable --now openvpn-client@farm.
Tailscale exit node: ts 1.98.9 in the container, node exit-vpn / 100.98.208.98, --advertise-exit-node --advertise-routes=192.168.105.0/24 --accept-dns=false, ip_forward via /etc/sysctl.d/99-tailscale.conf. Exit node approved in the admin console; the 192.168.105.0/24 subnet route is a separate checkbox and may still need ticking. Off by default — toggle with tailscale set --exit-node=exit-vpn / tailscale set --exit-node=.
tailscaled goes offline when the full tunnel rewrites the default route → fixed by tailscale-vpn-refresh.service (oneshot, PartOf/WantedBy=openvpn-client@farm.service, sleep 10; systemctl restart tailscaled).
FaceKom “VPN-only” hosts are NOT a DNS problem.npm.facekom.net resolves to the same public IP 92.119.122.189 from office DNS and 1.1.1.1; it sits behind an nginx source-IP ACL — 403 from the home WAN, 200 when sourced from the VPN. Routing, not name resolution.
Bundle + balazs’s screenshots live on the Mac at ~/telep/facekom-vpn/ (700/600). Both credential tars deleted from ~/Downloads. No secrets in this vault.
Repos / backup (NEW)
github.com/wowjeeez/telep-infra (PRIVATE) — infra snapshot: bin/ (camwall/pane/top4/agentctl scripts), services/ (intruder-alarm.py, top4-web.py), systemd/ units, nvr/ (sanitized configs). Secrets scrubbed (placeholders); rclone/keys/runtime-state git-ignored. Point-in-time — not live-synced; re-run the snapshot to refresh.
Obsidian vault auto-commits+pushes to github.com/wowjeeez/obsidian (the documenter does this).
OPEN / next
Power hardware (carry forward — all physical/BIOS access): BIOS Restore on AC Power Loss → Power On is the highest-leverage fix — auto-recovers the box from the daily ~06:00 mains blip like the router already does, removing the need for WOL / a physical press. Then a UPS + NUT for mains sags/graceful shutdown, and a bigger/better PSU for the load resets. Remote-wake (WOL) is NOT possible as configured (see the “Since last handover” block).
KB deferred: 27 image-only kick-fix PDFs still need an OCR pass; vectorization phase 2 (semantic dedup/cluster + explorer) is DONE (2026-07-24-kb-semantics-phase2-complete); phase 3 (claude -p consolidated generation — needs Claude Code installed + logged-in on the box, NOT yet done) not started (clusters.json is its retrieval bundle); optional bridge = a claude -p relabel behind make_label for nicer cluster titles; diagram understanding (wiring-diagram → structured/Mermaid via hybrid CV path-tracing + a vision model, accuracy-guarded, keep the original scan as source of truth) = a later phase.
Multi-pane control-remote (handover feature #2): a web page with pane buttons calling pane-switch, on its own ts-serve port. Not built. (Live screen-share was explored → HDMI-capture pane built → scrapped/reverted; panes are cctv+web only now.)
Persist /mnt/win via a systemd mount unit keyed on UUID= and ordered Before=docker.service — currently every reboot needs a manual mount plusdocker restart filebrowser.
filebrowser first-run / auth is no-login (tailnet-only) — fine as-is.
SECURITY — rotate these (values redacted here; they leaked earlier in transcripts + the obsidian repo history)
Tailscale API key (tskey-api-…) — revoke in admin console.
Telegram bot token — regenerate via BotFather if desired (used by frigate-notify + the alarm).
Frigate admin password (first-boot autogen) — change in UI.
Google Drive rclone OAuth — revoke at myaccount.google.com if desired.
Camera RTSP creds + telep-cc WiFi PSK also leaked.
NEW surface: the router LAN dropbear + ~/.ssh/router_alarm key (key-only, LAN-only — low risk, but note it exists).
⚠️ SESSION-HANDOVER.md and the vault are in the pushed wowjeeez/obsidian repo; keep literal secrets OUT of these notes going forward.
Key gotchas (hard-won)
⭐ On OpenWrt/busybox use setsid, not nohup (absent), to detach a self-reverting service swap. Keep a persisted binary backup on /overlay (survives reboot) and auto-revert if the new service isn’t online within N seconds — so a bad swap can’t lock you out of your only remote foothold. Used to upgrade the router’s tailscale 1.80.3→1.98.9 out-of-band; opkg still records 1.80.3-r1, so a future opkg upgrade could clobber the new binary (re-apply the static build or hold the package; backup at /overlay/ts-backup/). See the “Since last handover (2026-07-24)” block.
⭐ markdownify SILENTLY DROPS <img> inside <table> cells. Old sites use tables for LAYOUT, so nearly all figures get copied to disk but never referenced (11 refs for 122 pages initially). FIX: before markdownify, .unwrap() the layout containers (table/tbody/tr/td/th/center/font) so text+images flow linearly in doc order, rewrite each <img src> to the copied basename in place, wrap imgs in their own <p> (→ 528 refs). Keep a trailing ## Figures section only as a safety net. See 2026-07-24-kickfix-ingest.
⭐ “See photo here” TEXT links (<a href="X.jpg">) are separate from inline <img> and were 365/664 broken — their targets were never copied. FIX: also walk <a href>; if the href (URL-decoded, ../ resolved cross-topic) is an existing image, copy it under basename and rewrite href to basename (keep as a text link, don’t inline); leave external http(s)/mailto untouched; drop dead archive-local hrefs (→ 365→0 broken, images 449→741). See 2026-07-24-kickfix-ingest.
⭐ TWO separate power problems — don’t conflate them, and use the router-uptime discriminator. (a) Overnight MAINS outages: wall power drops, the router dies WITH the PC → fix is a UPS. (b) PSU-can’t-sustain-peak-load resets: wall power fine, PSU browns out under full load, box hard-resets while the router stays up 14h+ → fix is serialize jobs / cap threads / bigger PSU; a UPS does NOTHING for (b). Both show the same dead-stop journal signature — cross-check the router’s uptime to tell them apart. The default 16-thread CPU embedding maxing all cores reset the box in 2–6 min every time; OMP_NUM_THREADS=4/torch.set_num_threads(4) + nice + one heavy job at a time kept it up 45+ min and finished the index. See 2026-07-24-psu-load-resets.
⭐ HDBSCAN’s min_samples defaults to min_cluster_size — leaving it default made ALL section vectors noise (0 clusters). With min_cluster_size=2 and min_samples unset, HDBSCAN uses min_samples=min_cluster_size=2 and routes everything to the noise bucket. Set min_samples=1 explicitly for min_cluster_size=2 to actually form clusters. Looks like “the data has no structure” but is purely the default-coupling of the two knobs. See 2026-07-24-kb-semantics-phase2-complete.
⭐ Cross-source “span” is inflated when your id is <folder>/<manual> and one archive is split into many subfolders. The kick-fix archive is ONE source split into ~25 topic-subfolders, so ranking clusters by distinct manual_id made it look like ~25 manuals all covering a topic. Fix: define span by distinct TOP-LEVEL source folder (source_of = manual_id.split("/",1)[0]) → kick-fix collapses to 1 source while genuinely-separate manuals (suzuki-vitara/workshop vs /5door-supplement) still count. Keep source_manuals for display, rank by (span=distinct sources, #manuals, size). Also add a boilerplate label stoplist (is_boilerplate()) or cover/title pages (“SUZUKI”) and generic headings (“GENERAL DESCRIPTION”/“DIAGNOSIS”) sort to the top. And 51% noise singletons is EXPECTED/correct for a dedup goal — sections unique to one manual aren’t duplicated knowledge. See 2026-07-24-kb-semantics-phase2-complete.
The KB semantics explorer is a READ-ONLY route on the existing knowledgebase app, not a new service — the whole clustering pipeline only scrolls points OUT of Qdrant and writes a local clusters.json (never upsert/delete/create), and the label layer is pluggable behind make_label (future claude -p relabel). Deliberately no new always-on service on the power-flaky box. See 2026-07-24-kb-semantics-phase2-complete.
Always md5-dedup PDF downloads BEFORE (GPU-serialised) OCR — many were exact dupes or same-content-different-name (workshop ×2, 99501 ×2, parts-catalogue ×2); zips sometimes contain the PDF. See 2026-07-24-knowledgebase.
⭐ Debian 13’s stock initramfs (initramfs-tools) does NOT consume systemd-cryptenroll TPM2 tokens.systemd-cryptenroll --tpm2-device=… enrolls a valid TPM keyslot, but update-initramfs warns ignoring unknown option 'tpm2-device' and the box still prompts at boot — Debian’s classic cryptsetup hook can’t read a systemd-tpm2 token. For TPM LUKS auto-unlock on Debian use clevis (clevis-initramfs + clevis-tpm2, clevis luks bind … tpm2 '{}'), NOT systemd-cryptenroll + crypttab tpm2-device=. See 2026-07-24-luks-tpm-autounlock.
⭐ Adding ANY new LUKS keyslot needs the EXISTING passphrase — an agent cannot do it for the user. Both clevis luks bind and systemd-cryptenroll authorize the new slot against a current keyslot, so someone who knows the passphrase must type it. And the op is genuinely slow (argon2 KDF keyslot-add + TPM seal = tens of seconds) — an impatient Ctrl-C leaves a half-run process (which can orphan and hold the header lock). Let it finish. See 2026-07-24-luks-tpm-autounlock.
A stuck systemd-crypt* process (SSH killed mid-prompt) survives holding the LUKS header lock AND /dev/tpmrm0 and keeps re-broadcasting a systemd-ask-password request — silently blocking any later clevis luks bind (header locked). Fix: SIGKILL the orphan (SIGTERM may not clear one mid-TPM-op), confirm fuser /dev/nvme0n1p3 + lsof /dev/tpmrm0 clear and luksDump still shows the passphrase slot, before retrying. See 2026-07-24-luks-tpm-autounlock.
⭐ A separate tailnet identity on one host = a second tailscaled with --tun=userspace-networking, its own --socket and --statedir. Then tailscale --socket=… up --hostname=<name> (one interactive auth) + tailscale --socket=… serve …. Gives <name>.<tailnet>.ts.net as its own device without a container or GPU passthrough — the app + OCR stay on the host where the GPU + marker venv are. Used for the knowledgebase node — 2026-07-24-knowledgebase.
⭐ tailscale serve --https fails with “no TailscaleVarRoot” when the second tailscaled was started with only --state=<file>. Symptom is a TLS handshake error — curl gets 000 even with -k while TCP/ping succeed, so it looks like a listener/firewall issue but isn’t. Cause: with only a state file tailscaled has no var root to store the Let’s Encrypt cert. Fix: start it with --statedir=<dir> instead (the state file lives inside the statedir, so the node stays authenticated); then tailscale --socket=… cert <name> provisions the cert and TLS works — 2026-07-24-knowledgebase.
A journal that stops dead with no Stopping/panic/thermal trace = power was physically removed, not a software or OS shutdown. Software shutdown always logs Stopping; a panic always logs a trace — their absence is the diagnosis. See 2026-07-23-mains-power-shutdowns.
To tell a PC PSU fault from a mains event, cross-check an independent always-on device’s boot time (e.g. the router). Derive it from date +%s minus /proc/uptime. Same-timestamp death = mains; only-the-PC died = its own PSU.
A desktop that “turns off and stays off” while other gear silently recovers is the BIOS Restore on AC Power Loss = stay off policy, not a different failure. Embedded gear (router) auto-powers-on; the PC needs the policy flipped to match.
NVDEC on the 3080 corrupts camwall live feeds (magenta) under GPU load → --hwdec=no. Grab :0 framebuffer to actually see the TV.
Frigate birdseye is NVENC-encoded (h264_nvenc, inherits global hwaccel) → the encoder session can wedge and corrupt the wall’s live panes blue/purple even at GPU idle; docker compose restart frigate clears it, blank hwaccel_args for a durable libx264 fix. Isolate with: grab birdseye + source cam + /api/<cam>/latest.jpg — all-clean-but-birdseye = the encode stage. See 2026-07-18-birdseye-nvenc-blue-glare.
Never letterbox in the wall’s lavfi --vf (pad config-fail kills the composite) → letterbox canvas-side into 880×495 instead.
X-Forwarded-For is the LAST quoted field before request_time= in an nginx access line, not the first. Leftmost-match (re.search(r'"([^"]*)" request_time=') is spoofable: a client injecting a literal " into its User-Agent/Referer plants an earlier "…" request_time= occurrence and forges the captured viewer IP. Default nginx escapes quotes so it’s safe in practice, but take findall(...)[-1] (the field truly adjacent to request_time=) so you don’t depend on the log-escape setting. Applies to ANY nginx-log parsing that trusts a quoted field — 2026-07-23-frigate-viewer-alert.
Never spawn tailscale status (or any ~10s subprocess) per log line. Refreshing a name cache on every unknown-IP line becomes one blocking subprocess PER line from an active viewer when tailscale status is momentarily failing or the IP isn’t listed — a self-DoS. Rate-limit miss-triggered refreshes (≤ once/60s) on top of the periodic refresh — 2026-07-23-frigate-viewer-alert.
mpv destroys osd-overlay entries the moment the IPC client disconnects — connect→send→close silently does nothing (mpv still replies {"error":"success"}). The IPC connection must be persistent, and you must drain mpv’s event stream or the socket buffer fills. Reconnect-on-failure is what makes the overlay self-heal after a camwall restart.
camwall-reload is unusable for live data — its loadfile restarts video playback (fine weekly for TOP KÉPEK, would stutter the live cameras every 5s). Use osd-overlay over /run/camwall.sock: touches neither the stream nor the filter chain. Note /run/camwall.sock is srw------- root root → overlay pushers must run as root.
iwinfo assoclist has no byte counters; iw dev <ap> station dump does (rx/tx bytes + signal, iw already installed) → per-device bandwidth needs no nftables/conntrack/nlbwmon.
Station byte counters are from the AP’s point of view: tx bytes = the device’s DOWNLOAD, rx bytes = its upload. The WAN interface is NOT inverted. Easiest bug to ship.
SSH ControlMaster masks credential loss — deleting the router key changes nothing while the multiplexed ControlPersist connection lives (the key file is never re-read). To test degradation you must also drop the control socket (/run/wifi-usage/router.sock).
Sleeping iOS phones drop ARP but stay WiFi-associated → presence needs the router assoc table, not just arp-scan.
Camera VLAN AP (phy1-ap1/telep-cc) leaks the camera into any assoc-based device list → exclude it.
Telegram sends take ~15s on the flaky WAN → answer callbacks FIRST, never hold a lock during network I/O.
dig/subprocess error text can leak into data if stdout isn’t validated → strict hostname regex + stderr to devnull.
iOS AirPrint can’t work over Tailscale (mDNS/L3). Mac-by-IP works.
macOS: bare ssh broken (use /usr/bin/ssh); iCloud Private Relay breaks tailnet MagicDNS in iOS Safari.
NVMe device nodes on the mainframe flip across reboots (nvme0n1 ↔ nvme1n1) → lsblk -f before mounting, UUID= in fstab.
A docker bind-mount captures the host dir at container start — mounting a filesystem under it afterwards leaves the container with an empty stale view. Mount first, then (re)start the container.
Two OpenVPN servers live on 92.119.122.0/24 — connecting to the wrong one (.185 vs .32) presents a different CA and looks exactly like a “bad CA bundle”. Verify the gateway IP before blaming certs.
Vendor auth.txt is labelled + tab-delimited (user: X, jelszó:<TAB>pw) → the auth-user-pass file must be exactly 2 bare lines or OpenVPN sends the label as part of the password (AUTH_FAILED).
openvpn-client@.service is hardened with a process limit → any up/route-up script fails with unable to fork … (errno=11) and kills the tunnel; use a separate systemd unit instead.
Tailscale exit nodes don’t carry RFC1918 — private LAN access needs an approved subnet route, not just the exit node.
Full-tunnel route changes strand tailscaled’s control connection → node shows offline; restart tailscaled after the tunnel comes up.
When testing a route change, curl may resolve a different (anycast) IP than the one you routed → always pin with curl --resolve host:443:<IP>.
TensorRT dead on amd64 (0.17)→ONNX; Tapo truncates >14-char pw, Basic auth only, reboot after cred change; /cam1+/cam2 RTSP = same lens (use /stream1 vs /stream6); Docker FORWARD=DROP breaks incus → DOCKER-USER fix (agent-net-fix.sh).