Levandor

Homelab Activity Log

24 min read

For Agents

Reverse-chronological session log. Newest entries at top, grouped by date (## YYYY-MM-DD). Each bullet: one piece of work, short summary, wikilinks to docs touched. Updated by obsidian-documenter on every project doc write. Read by historian at bootstrap.

2026-07-24

  • KB semantics phase 2 COMPLETE — fully-local semantic SECTION clustering over the phase-1 vectors + a read-only web explorer (“Szemantika”), NO LLM. Pipeline: scroll all 3252 chunk vectors from Qdrant manuals (READ-ONLY) → group by (manual_id,page) → L2-normalized mean-pool → ~1248 SECTION vectors → HDBSCAN(min_cluster_size=2, min_samples=1, euclidean≈cosine) → representative-heading labels → clusters.json. 194 clusters (post boilerplate filter), 24 cross-SOURCE, 631 noise singletons (noise EXPECTED for a dedup goal). Top clusters are real cross-make overlaps (WINDOW REGULATOR spanning chevy-tracker + suzuki-vitara/5door + workshop; ENGINE COOLING, PARKING BRAKE, THROTTLE BODY). Code /home/levander/kb-vectors/ (sectionvecs.py/labels.py/cluster.py/kbclust.py + tests, CLI build/list/show); explorer /semantics + /api/clusters folded into the EXISTING knowledgebase Flask app (NO new service) at knowledgebase.taild4189d.ts.net/semantics; 🧭 Szemantika dashboard tile. Ranked by cross-SOURCE span so dedup targets surface first; labels pluggable behind make_label for a future claude -p relabel. Phase 3 (claude -p generation) still NOT started — clusters.json is its retrieval bundle — 2026-07-24-kb-semantics-phase2-complete
  • ⭐ Gotcha — HDBSCAN’s min_samples defaults to min_cluster_size, so leaving it default made ALL sections noise (0 clusters). Must set min_samples=1 explicitly for min_cluster_size=2 to form clusters — looked like “no structure in the data” but was purely the two knobs being coupled by default — 2026-07-24-kb-semantics-phase2-complete
  • ⭐ Gotcha — cross-source span was inflated because manual_id is <folder>/<manual> and the kick-fix archive is ONE source split into ~25 topic-subfolders (each counted as a distinct “manual”). Fix: define span by distinct TOP-LEVEL source folder (source_of = manual_id.split("/",1)[0]) so kick-fix collapses to 1 source while genuinely-separate manuals still count; keep source_manuals for display, rank by (span=distinct sources, #manuals, size). Added an is_boilerplate() label stoplist (22 clusters dropped — cover pages “SUZUKI”, “GENERAL DESCRIPTION”/“DIAGNOSIS”). 51% noise is EXPECTED/correct for dedup. Whole pipeline is READ-ONLY scroll → local clusters.json, never upsert/delete — 2026-07-24-kb-semantics-phase2-complete
  • Updated SESSION-HANDOVER (new Szemantika/semantics service section, phase-2-DONE + phase-3-still-open, the two starred min_samples + span-inflation gotchas in Key gotchas, OPEN/next KB-deferred) and TOPICS
  • Live state ~14:00: box UP (booted 13:55 after a physical power-on following a morning mains outage); clevis TPM auto-unlock worked again unattended; all services active (knowledgebase, home-portal, wifi-usage, intruder-alarm, camwall, both tailnet nodes, kb-qdrant + frigate Up); Qdrant manuals at 2839 points — SESSION-HANDOVER
  • Upgraded telep-router Tailscale 1.80.3 → 1.98.9 — hand-installed a static arm64 build (the opkg feed only carries 1.80.3) via a setsid-detached self-reverting swap script that rolls back to /overlay/ts-backup/ if the new binary isn’t online in 45s. Router online, 8 peers, verified over SSH served by the new binary. ⚠️ opkg still RECORDS 1.80.3-r1 → a future opkg upgrade could clobber the new binary. Gotcha: on OpenWrt/busybox use setsid not nohup (absent) + a persisted /overlay backup + auto-revert so a bad swap can’t lock you out of your only remote foothold — Since last handover (2026-07-24)
  • ⚠️ New observation — a daily ~06:00 mains die-off pattern. The box (and the router with it) appears to lose power around 06:00 most mornings (Jul 19 ~06:59, Jul 23 ~06:36, Jul 24 morning). The router dies too = MAINS-level, distinct from the PSU-load resets (2026-07-24-psu-load-resets). Suspect a scheduled utility relay (Hungarian vezérelt/GEO off-peak tariff) or a timed high-draw appliance browning the shared circuit — NOT confirmed; confirm via journalctl --list-boots vs router-reboot regularity — Since last handover (2026-07-24)
  • Remote-wake (WOL) is NOT possible as configured — after a mains outage the box needs a physical power-button press; WOL never enabled (BIOS+NIC), the router has no etherwake/wakeonlan, and the box’s MAC is unknown after the DHCP lease expires (no static reservation). Better fix than WOL: BIOS Restore on AC Power Loss → Power On so the box auto-recovers from the daily blip like the router (one physical BIOS visit) — Since last handover (2026-07-24)
  • In-flight (check next session): ~/ocr/resume_extras.sh (setsid-detached, sentinels .done/.failed) — body-measurements already OCR’d+seeded into suzuki-vitara; this OCRs the Sidekick parts-catalogue → seeds suzuki-sidekick/parts-catalogue-89-98 → rebuilds the site → vectorizes BOTH new manuals into Qdrant (thread-capped OMP_NUM_THREADS=4 + nice). Re-run bash ~/ocr/resume_extras.sh if it died — Since last handover (2026-07-24)
  • Updated SESSION-HANDOVER with a “Since last handover” delta block (router tailscale upgrade, daily-06:00 mains pattern, WOL, in-flight OCR job) + a live-state callout, added the ⭐ setsid-not-nohup gotcha, the daily-mains and opkg-clobber warnings, and carried the power-hardware + KB-deferred items into OPEN/next
  • Expanded the knowledgebase manual library — many OCR’d manuals now under docs/<folder>/<manual>/: suzuki-vitara (5door-supplement, supplement-61a40/99501, workshop-1988-1998 835pg, wiring-diagrams/99512, body-measurements), chevy-tracker (owners-1995, geo-tracker-repair), suzuki-sidekick (wiring-1996, parts-catalogue-89-98). OCR pipeline marker (~/ocr/venv, marker_single --force_ocr, TORCH_DEVICE=cuda) → seed_import.py → build; manuals queued as CHAINED nohup scripts (each waits on the prior’s .done sentinel) to serialize the single GPU. Dedup discipline: md5-dedup downloads BEFORE OCR — many were exact dupes / same-content-different-name (workshop ×2, 99501 ×2, parts-catalogue ×2); zips sometimes contain the PDF (sidekick wiring was a zip of wiring2.pdf) — Library expansion (2026-07-24)
  • Ingested the kick-fix.com archive — a 112 MB offline archive of the defunct kick-fix.com Suzuki Sidekick/Tracker repair site into docs/kick-fix/ (25 topics, 127 articles, 741 images, 8 text-layer PDFs; 27 image-only PDFs deferred for OCR). Dead-site archive → no ToS/robots issue. Script /home/levander/knowledgebase/kickfix_ingest.py (KB venv + markdownify + beautifulsoup4 + pypdfium2) — 2026-07-24-kickfix-ingest
  • ⭐ Gotcha — markdownify SILENTLY DROPS <img> inside <table> cells. Old sites use tables for LAYOUT so nearly all figures were copied to disk but never referenced (11 refs for 122 pages). FIX: before markdownify, .unwrap() the layout containers (table/tbody/tr/td/th/center/font) so text+images flow linearly, rewrite <img src> to the copied basename in place, wrap imgs in own <p> (→ 528 refs); keep a trailing ## Figures only as a safety net — The two that cost real time
  • ⭐ Gotcha — “See photo here” TEXT links (<a href="X.jpg">) are separate from inline <img>, 365/664 broken (targets never copied). FIX: also walk <a href>; if the URL-decoded, ../-resolved href is an existing image, copy under basename + rewrite href (keep as text link, don’t inline); leave external http(s)/mailto; drop dead archive-local hrefs (→ 365→0 broken, images 449→741). src/href need URL-decode (%20) + cross-topic ../ resolution — The two that cost real time
  • Vectorization phase 1 COMPLETE — built /home/levander/kb-vectors/ (chunker/embed/index/search/kbvec CLI): local BAAI/bge-large-en-v1.5 CPU embeddings + Qdrant container kb-qdrant (127.0.0.1:6333, collection manuals, 1024-dim cosine), idempotent per-manual (delete-by-filter + uuid5 ids). 2839 chunks indexed across 33 manual/kick-fix collections; cross-source semantic search blends official manuals + kick-fix community guides. Chose Qdrant over embedded stores for payload filtering + UI + growth; suzuki-forums crawler DROPPED (Tollbit-gated). Foundation for phase 2 (semantic dedup) + 3 (claude -p generation) — 2026-07-24-kb-vectorize-complete
  • ⭐ Root-caused a NEW, distinct power problem: PSU-can’t-sustain-peak-load resets (separate from the overnight mains outages). The box crash-rebooted ~3× in 30 min (04:11/04:20/04:32) under many heavy jobs at once (GPU OCR + CPU embedding + kick-fix conversion + Frigate). NOT thermal (CPU 39°C/GPU 47°C, no MCE), NOT mains — the router stayed up 14h+ straight through (the router-uptime cross-check is the discriminator; mains = router dies too). Dead-stop journal signature. The default 16-thread CPU embedding maxing all cores reset the box in 2–6 min every time. MITIGATION: cap to 4 threads (OMP_NUM_THREADS=4/torch.set_num_threads(4)) + nice + serialize heavy jobs → box stayed up 45+ min, finished the index. A UPS does NOT fix this; real fix = a bigger/better PSU — 2026-07-24-psu-load-resets
  • clevis LUKS TPM2 auto-unlock VALIDATED IN PRODUCTION — the PSU reboot storm above doubled as the reboot test: the box self-recovered the encrypted root through all 3 unattended reboots (systemd-cryptsetup@nvme1n1p3_crypt on boot, root mounted, no passphrase). All services auto-recovered (knowledgebase, kb-qdrant, home/knowledgebase tailscale nodes, Frigate, camwall, wifi-usage, intruder-alarm). Reboot-test status: pending → PASSED in production — Validated in production (2026-07-24 reboot storm)
  • Added a global dashboard home.taild4189d.ts.net (own tag:telep node, Hungarian “Elérhető rendszerek”, tiles Kamerák/Tudásbázis/Top képek/Router); the knowledgebase also became its own tag:telep node. Same reusable pattern: a dedicated second tailscaled per identity (--tun=userspace-networking, own --socket/--statedir) — 2026-07-24-global-dashboard
  • Updated SESSION-HANDOVER (KB expansion + kick-fix + kb-vectors/Qdrant + global-dashboard sections, PSU-load power bullet, LUKS reboot-test PASSED) and added the starred markdownify-table-images, text-link-images, PSU-vs-mains and md5-dedup gotchas
  • Set up TPM2 auto-unlock for telep-mainframe’s root LUKS — the follow-up to the mains-power work so an unattended reboot no longer stalls at the LUKS passphrase prompt (unreachable over SSH, no dropbear-initramfs). Root is LUKS2-on-LVM (/dev/nvme0n1p3, mapper nvme1n1p3_crypt, UUID a3a8e37d-79fa-484b-bc3f-40c56df95337); TPM2 /dev/tpmrm0 (MSFT0101), Secure Boot OFF. Bound the TPM with clevis (clevis-initramfs+clevis-tpm2, clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}', empty PCRs) → keyslot 1 = TPM, keyslot 0 = passphrase kept as fallback; update-initramfs -u installs the clevis unlock hook so the disk auto-unseals at boot. Header backed up first at ~/telep/luks-header-nvme0n1p3-20260723.img — 2026-07-24-luks-tpm-autounlock
  • Gotcha — Debian 13 stock initramfs (initramfs-tools) does NOT consume systemd-cryptenroll TPM2 tokens. Tried systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs= first: it enrolls a valid TPM keyslot, but update-initramfs warns ignoring unknown option 'tpm2-device' and the box still prompts at boot (the classic cryptsetup hook can’t read a systemd-tpm2 token). Abandoned that path (wiped the slot with --wipe-slot=tpm2, no passphrase needed) and used clevis instead — [[2026-07-24-luks-tpm-autounlock#why-not-systemd-cryptenroll|Why not systemd-cryptenroll]]
  • Gotcha — adding any new LUKS keyslot needs the existing passphrase (an agent can’t do it for the user) and is genuinely slow (argon2 KDF + TPM seal, tens of seconds); an impatient Ctrl-C orphans a systemd-crypt* process that SURVIVES holding the LUKS header lock AND /dev/tpmrm0 and re-broadcasts a systemd-ask-password — silently blocking the later clevis luks bind. Fix: SIGKILL the orphan, confirm fuser/lsof clear + passphrase slot intact, retry — Hard-won gotchas
  • Honest incomplete state: reboot test still PENDING (needs console access — a failed unlock hangs with no SSH); auto-unlock only helps once the box BOOTS, and BIOS Restore on AC Power Loss is still stay-off so a mains cut still needs a button press (2026-07-23-mains-power-shutdowns); empty-PCR + Secure-Boot-off is an accepted tradeoff — protects against bare-drive theft, not whole-box theft. A passphrase-echo hygiene issue occurred → passphrase rotation suggested (no values in the vault) — complementary (honest state)
  • Added a “Disk encryption / auto-unlock” note to the drives section and the two starred TPM/keyslot gotchas to SESSION-HANDOVER; cross-linked the follow-up from 2026-07-23-mains-power-shutdowns
  • Built the knowledgebase — a tailnet-only manual library + OCR ingest portal on telep-mainframe, live at https://knowledgebase.taild4189d.ts.net as its own tailnet node (not a port on the mainframe). Renders OCR’d PDF manuals as an mkdocs-material static site (folder-tree browse, full-text search, dark mode, phone-responsive, all images loading="lazy") plus a Flask+waitress /upload portal that runs marker OCR on the GPU, splits the markdown into per-section pages, copies images and atomically rebuilds the site. Two enabled systemd units (knowledgebase.service app on 127.0.0.1:8092 as levander, tailscaled-knowledgebase.service the dedicated node). Code /home/levander/knowledgebase/ (26 tests). Seeded with the Suzuki Vitara 5-door supplement (59 pages/1248 imgs) + Chevy Tracker 1995 owners manual (5 pages/352 imgs) — 2026-07-24-knowledgebase
  • Core pattern — a separate tailnet identity on one host = a second tailscaled with --tun=userspace-networking, its own --socket/--statedir, then tailscale --socket=… up --hostname=knowledgebase (one interactive auth) + tailscale --socket=… serve. Gives knowledgebase.<tailnet>.ts.net as its own device without a container or GPU passthrough — the app + OCR stay on the host where the GPU + ~/ocr marker venv live — The two that cost thought (separate-node + TLS)
  • TLS gotcha — tailscale serve --https fails with “no TailscaleVarRoot” when the second tailscaled is started with only --state=<file>: a TLS handshake error where curl gets 000 even with -k though TCP/ping succeed. tailscaled has no var root to store the LE cert. Fix: start it with --statedir=<dir> instead (the state file lives inside the statedir, so the node stays authenticated), then tailscale --socket=… cert <name> provisions the cert — The two that cost thought (separate-node + TLS)
  • Chunked loading = split each manual by its # h1 headings into many small per-section HTML pages (a 795 KB markdown would be one huge page) + a mkdocs on_post_page hook injecting loading="lazy" into every <img> (verified 1600/1600 lazy) — this fixed “Ulysses chokes on 1248 images”. Browse is the folder tree under manuals-src/docs/ → mkdocs auto-nav (no hand-maintained nav:); one service serves both the static site and the /upload portal; mkdocs builds to a temp dir then os.rename-swaps site/ so a broken source never takes down the live site — How it works (the reusable parts)
  • Ingest reuses the existing ~/ocr/venv marker pipeline as a subprocess (marker_single, TORCH_DEVICE=cuda) — no duplicate torch/model install; single background worker, GPU-serialised (shared with Frigate); job states queued→ocr→splitting→building→done/failed. Path-traversal guard (safe_slug + realpath containment) before any write, %PDF magic-byte check, 500 MB cap. Re-ingesting the same folder+name must rmtree the destination first or stale orphan pages/images linger as ghost pages in nav/search — Security & robustness
  • Added the Knowledgebase section and the two starred tailnet-node/TLS gotchas to SESSION-HANDOVER

2026-07-23

  • Built the Frigate viewer alert — frigate-viewer-alert.service on telep-mainframe (python3 stdlib, /home/levander/frigate-viewer-alert/, 34 tests, runs as levander not root): follows docker logs -f --since 0m frigate, parses each nginx access line, reads the real client IP from X-Forwarded-For (set by tailscale serve), maps 100.x → tailscale device name and Telegrams 👁 <device> (100.x) opened Frigate — HH:MM[ · live: <cam>] on a new UI session. Reuses the frigate-notify bot+chat (token/chatid read at runtime from ~/nvr/frigate-notify/config.yml, never copied) — viewer pings land beside the person/car alerts — 2026-07-23-frigate-viewer-alert
  • Core detection technique: tailscale serve writes the tailnet client IP into the nginx X-Forwarded-For field. A real viewer → 100.x XFF; internal automation (frigate-notify polling, self-polls) → XFF -, excluded by construction with no allowlist. Both live (/live/jsmpeg/… 101) and plain UI (/api/config, /login) requests carry it, so every session is visible off one log stream. “Every UI session without spam” via a per-source-IP 5-min cooldown — an open tab polls constantly so it edge-triggers (first request after a quiet gap = new session; later requests just refresh the timer) — How detection works (the reusable part)
  • Security gotcha: X-Forwarded-For is the LAST quoted field before request_time=, not the first. Leftmost-match (re.search) is spoofable — a client injecting a literal " into its User-Agent/Referer forges an earlier "…" request_time= and plants the captured viewer IP. Default nginx escapes quotes so it’s safe in practice, but findall(...)[-1] (the field truly adjacent to request_time=) removes the dependency on the log-escape setting. Reusable for any nginx-log parsing that trusts a quoted field — [[2026-07-23-frigate-viewer-alert#1-x-forwarded-for-is-the-last-quoted-field-before-request_time|1. X-Forwarded-For is the LAST quoted field before request_time=]]
  • Ops gotcha: never spawn tailscale status per log line. An early version refreshed the name cache on every unknown-IP line; if tailscale status is momentarily failing or the IP isn’t listed, that’s one blocking ~10s subprocess PER line from an active viewer — a self-DoS. Rate-limit miss-triggered refreshes (≤ once/60s) on top of the periodic refresh. Also: reset the docker logs -f reconnect backoff only after a line is actually read, or an immediate-EOF crashloop never escalates; --since 0m so restarting the daemon doesn’t replay stale alerts — [[2026-07-23-frigate-viewer-alert#2-never-spawn-tailscale-status-per-log-line|2. Never spawn tailscale status per log line]]
  • Added the Frigate viewer alert bullet under Frigate NVR and gotchas 1+2 to SESSION-HANDOVER
  • Root-caused telep-mainframe’s recurring “randomly turns off overnight” to MAINS POWER loss, not the PC. Recurring unclean shutdowns in last -x (Jul 23 06:36, 19 06:59, 17 21:28, 16, 14); the box stays dark until manually powered on. Verdict conclusive — the telep-router (separate hardware/PSU) died at the same instant (mainframe journal dead 06:36:10 CEST, router booted 06:36:27, +17s POST), which only a shared-mains event can cause — 2026-07-23-mains-power-shutdowns
  • Reusable method — the dead-stop signature: journalctl -b -1 ends mid-normal-operation (a routine arp-scan presence check) with NO Stopping sequence, NO panic/stack trace, NO thermal/MCE/Xid line, NO OOM. That absence is the diagnosis: software shutdown always logs Stopping, a panic always logs a trace → a clean dead-stop = power physically removed. Confirmed cool (acpitz 27°C, GPU 47°C/128W), no poweroff/suspend timers, atq/wakealarm empty, sleep+suspend masked — Evidence chain (the reusable method)
  • Clincher method — cross-check an independent always-on device: derive the router’s boot epoch from date +%s minus /proc/uptime and compare to the instant the PC’s journal died. Same-timestamp death = mains; only-the-PC = its own PSU. Turns “probably power” into “definitely mains” — Evidence chain (the reusable method)
  • Explained why only the PC “stays off”: BIOS Restore on AC Power Loss = stay-off (common default) so the desktop sits dark until someone presses power, while the router auto-powers-on (embedded, no button) so its recovery is invisible. Every last -x crash → a boot many hours later, never an auto-restart (inconsistent with panic=reboot) — Why only the PC appears to “stay off”
  • Recommendations to the user, priority order: (1) ~600–900 VA UPS + NUT — the real fix for unstable mains, logs every sag/transfer and triggers graceful shutdown on long outages; (2) free stopgap — flip BIOS AC-loss to Power On (physical BIOS access only, not over SSH); (3) rule out an overloaded shared circuit (i9-12900K + RTX 3080 ~500W + 4K TV). ext4 clean and all services recovered so far, but repeated hard cuts risk eventual corruption. Noted frigate.service reads inactive only because Frigate is docker compose, not a systemd unit (containers healthy) — Recommendations (priority order)
  • Added the mains-power bullet under Hardware/host and the three reusable lessons to SESSION-HANDOVER

2026-07-22

  • Built the WiFi device usage strip on the camwall — wifi-usage.service on telep-mainframe (python3 stdlib, /home/levander/wifi-usage/, 44 tests): every 5s it polls the router over one persistent SSH connection, diffs per-station byte counters into live Mbps, resolves MACs to names and pushes an ASS osd-overlay into the running mpv. Strip along the bottom of the cctv pane (y 1880–2140) showing every telep1 device’s ↓/↑ plus a 5G WAN total — 2026-07-22-wifi-usage-strip
  • Core discovery: mpv destroys osd-overlay entries when the IPC client disconnects. Connect→send→close silently does nothing while still replying {"error":"success"}; holding the same socket open with an identical payload made the strip appear. So the IPC connection must be persistent, mpv’s event stream must be drained, and reconnect-on-failure is what gives self-healing (strip returned ~23s after a camwall restart) — 1. The mpv IPC connection must be persistent
  • iwinfo assoclist has no byte counters; iw dev <ap> station dump does (rx/tx bytes + signal, iw already on the router) → per-device bandwidth with no extra package, no nftables accounting, no conntrack parsing, no nlbwmon. And the counters are AP-perspective: station tx bytes = the device’s DOWNLOAD, rx bytes = its upload; the WAN interface is not inverted (verified live: device 35.4 ↑ vs WAN 35.1 ↑) — [[2026-07-22-wifi-usage-strip#2-bytes-come-from-iw-not-iwinfo|2. Bytes come from iw, not iwinfo]]
  • Chose osd-overlay over camwall-reload: camwall-reload’s loadfile restarts video playback (fine weekly for TOP KÉPEK, would stutter the live cameras every 5s) and goes through the lavfi chain that previously killed the wall. Also: /run/camwall.sock is srw------- root root → the daemon must run as root — [[2026-07-22-wifi-usage-strip#4-osd-overlay-never-camwall-reload|4. osd-overlay, never camwall-reload]]
  • Security work on an untrusted input path (DHCP hostnames are attacker-supplied): subprocess.run(text=True) decodes strictly and UnicodeDecodeError is a ValueError not caught by except (SubprocessError, OSError) → errors="replace"; ASS escaping stops tag injection (validated vs libass 0.17.3) but names still need a 22-char cap or a 400-char hostname paints across the 4K frame; camera VLAN excluded by construction (phy1-ap1 never queried) — Untrusted input: LAN device names
  • Gotcha: SSH ControlMaster masks credential loss — removing the router key showed no degradation because the multiplexed ControlPersist connection was already authenticated and the key file is never re-read; you must also drop /run/wifi-usage/router.sock to test it — SSH ControlMaster hides credential loss
  • Added the WiFi usage strip section and five new gotchas to SESSION-HANDOVER
  • Unstalled the FaceKom “VirtualOffice” OpenVPN on the exit-vpn container and wired it as an on-demand Tailscale exit node (exit-vpn / 100.98.208.98). The CA was never broken — real causes were the wrong gateway IP (.185 vs 92.119.122.32, both running OpenVPN on tcp/1194 → foreign CA), wrong crypto (AES-256-CBC/SHA1), and an auth file still carrying the Hungarian jelszó:<TAB> label inside the password. Full tunnel via redirect-gateway def1, persistent via openvpn-client@farm — 2026-07-22-facekom-vpn-exit-node
  • Discovered FaceKom “VPN-only” hosts (npm.facekom.net) are public IPs behind an nginx source-IP ACL, not a DNS issue — 403 from home WAN, 200 via VPN, same resolution from every resolver. Also: route-up scripts are impossible under the hardened openvpn-client@.service (fork errno=11 kills the tunnel) → separate oneshot unit restarts tailscaled instead — Gotcha: tailscaled goes offline after VPN route changes
  • Corrected the now-wrong “exit-vpn — STALLED” section and added 6 new gotchas to SESSION-HANDOVER
  • Diagnosed an unplayable 1.6 GB SplitCam recording (20251025075725.mp4) pulled off the mainframe’s Windows drive via filestash: moov atom not found, mdat size 0. Verdict = unrecoverable — SplitCam crashed and NTFS-preallocated clusters were never flushed, so the mdat is stale deleted-file residue (7.997 bits/byte entropy, zero valid NAL chains, tail all zeros). Only ~69 KB of real H.264 hit disk — mp4-missing-moov-forensics
  • Reusable trick: xattr -p com.apple.metadata:kMDItemWhereFroms <file> | xxd -r -p | strings recovered the Filestash origin URL, which let the source at /mnt/win/Users/kisapus69/odi/data/SplitCam/ be chunk-md5-compared against the local copy (identical — source equally dead, no other copies) — Verifying the local copy wasn’t the problem

2026-07-21

  • Remounted the Windows NTFS drive at /mnt/win after a reboot and restarted filebrowser; browsing verified working at https://telep-mainframe.taild4189d.ts.net:8445 — Mount procedure (verified 2026-07-21)
  • Correction: the NVMe device path is NOT stable across reboots. The docs said /dev/nvme1n1p2; the Windows NTFS partition is now /dev/nvme0n1p2 (1.8T nvme0n1), and nvme1n1 is the 3.6T LUKS+LVM OS disk. Always lsblk -f and pick the 1.8T ntfs partition — and any fstab/systemd entry must use UUID=, not a device node — The Windows drive
  • Gotcha: filebrowser needs a restart after the mount. The container binds /mnt/win → /srv RO; started against an empty unmounted dir it holds the stale pre-mount view and /srv looks empty inside the container while the host mount is fine. docker restart filebrowser. Every reboot = mount, then restart, in that order — The Windows drive
  • Strengthens the open item on persisting /mnt/win: a systemd mount unit on UUID= ordered Before=docker.service would eliminate both the reboot gap and the container-restart dance — SESSION-HANDOVER
  • Drive inventory update: sdb is now present — 115.5G UDF labelled HBCD_PE_x64 (Hiren’s BootCD PE rescue USB stick); sda (447.1G) remains unused (single 16M partition, no filesystem) — Other services on this box

2026-07-18

  • Diagnosed & fixed the TV camwall “blue glare”: blocky blue/purple/green chroma on the two live panes (clean on the static JPEG panes). Root cause = a wedged NVENC encoder session on the RTX 3080 corrupting Frigate’s birdseye restream (-c:v h264_nvenc, inherited from global hwaccel_args: preset-nvidia) even at GPU idle — the encode-side sibling of the old NVDEC magenta bug — 2026-07-18-birdseye-nvenc-blue-glare
  • Reusable isolation method: grab a still at each pipeline layer (TV framebuffer via x11grab → birdseye RTSP → source cam RTSP → /api/<cam>/latest.jpg → ps aux | grep birdseye). source clean + latest.jpg clean + birdseye corrupt = the h264_nvenc encode is the sole corrupting stage — Isolation method (reusable — this is the valuable part)
  • Fix: cd ~/nvr && docker compose restart frigate cleared the wedged session (still on NVENC). Durable option if it recurs: blank hwaccel_args → birdseye encodes libx264 on the 12900K (YOLO stays on GPU; CUDA ≠ NVENC/NVDEC) — Durable fix if it recurs (not yet applied)
  • OCR’d a 564-page scanned Suzuki Vitara manual on the mainframe’s RTX 3080 via marker-pdf (~22.5 min, 794 KB markdown, 1,248 images) — set up a reusable venv at ~/ocr/venv. Core gotcha: pip install marker-pdf pulls a cu130 torch that driver 550.163.01 can’t init, so torch.cuda.is_available() is False and marker silently falls back to CPU (hours instead of minutes). Fix = pip uninstall -y torch then reinstall from the cu126 index (minor-version compat); the reinstall is a silent no-op without the uninstall. Driver deliberately NOT upgraded — 550 is load-bearing for Frigate — Running it on a CUDA GPU box (telep-mainframe), [[telep-mainframe#gpu-ocr-venv-ocrvenv|GPU OCR venv (~/ocr/venv)]]
  • Generic ops trap surfaced twice this session: pgrep -f / pkill -f match your own command line. The OCR completion monitor reported RUNNING for 2.5 h after the job finished; a pkill -f openvpn... over SSH killed its own shell (exit 143) and silently skipped the remaining steps — shell-gotchas-pgrep-self-match

2026-07-17

  • Built and documented the presence-based intruder alarm (intruder-alarm.service, /home/levander/intruder-alarm.py, python3 stdlib-only) on telep-mainframe: auto-arms when no trusted phone is on the network, escalates person/car detections to Telegram, sirens the LG TV — 2026-07-17-intruder-alarm
  • Presence = union of router WiFi association table (SSH iwinfo assoclist on telep1 APs) + arp-scan on enp5s0. KEY INSIGHT: sleeping iOS phones stay WiFi-associated but stop answering ARP, so arp-scan alone false-arms overnight; the assoc table sees them. Failed scans are skipped, never treated as “everyone left” — Presence detection
  • Gave the mainframe router access it never had (router was Tailscale-SSH-only, ACL-blocked, no port-22 daemon): enabled a LAN-only key-only dropbear (uci Interface=lan, PasswordAuth off) + dedicated key /home/levander/.ssh/router_alarm. GOTCHA: this rotated the router’s SSH host key (breaks existing known_hosts) — Prerequisite router access
  • Camera-VLAN gotcha: SSID telep-cc is AP phy1-ap1; querying it made the Tapo camera appear as an enrollable device. Fix: only query telep1 APs + exclude infra MACs (mainframe/printer/router/camera) — Camera-VLAN gotcha — telep-cc leaked an enrollable device
  • Enrollment via Telegram Trust/Ignore inline buttons (same “Dezsi az őr” bot as frigate-notify, token read from ~/nvr/frigate-notify/config.yml); device names resolved mDNS avahi-resolve → dnsmasq reverse DNS → eszköz-<last4>, hardened against leaking dig errors — Device naming & enrollment
  • State machine: dormant until ≥1 trusted device, arms after ALL trusted absent 10 min, disarms instantly on return, manual /arm /disarm /auto; intruder escalation 🚨 BETŐRŐ! + snapshot (1×day/2×night); TV siren (2-beep, aplay -D plughw:1,3, NVIDIA HDMI = ALSA card 1 dev 3) on every detection armed or not — State machine

2026-07-16

  • Deployed Filestash on telep-mainframe to browse the box’s Windows NTFS drive in-browser (previews for images/video/PDF/code/markdown/archives); container bound to 127.0.0.1:8334, config in filestash_data volume — filestash
  • Mounted Windows C: (/dev/nvme1n1p2, NTFS) read-only at /mnt/win via in-kernel ntfs3 (RO because the live Windows install’s fast-startup/hibernation would corrupt a RW mount); passed into the container at /mnt/data. Gap: mount is manual, not in fstab — won’t survive reboot — The Windows drive
  • Exposed it at https://telep-mainframe.taild4189d.ts.net:8445 via tailscale serve --https=8445, same pattern as top4 picker (:8443) and Frigate (:443); port map now :443/:8443/:8445 — Exposure
  • First-run wizard (admin password + “Local” backend at /mnt/data) left for the user to finish; OnlyOffice for inline Office docs noted as optional/not-added — First-run setup (unfinished)

2026-07-14

  • Built and documented the Frigate NVR end-to-end on bare-metal telep-mainframe (i9-12900K / RTX 3080 / Debian 13): Frigate 0.17.2 + frigate-notify, ONNX YOLOv9-s detection (~12ms, 6% GPU), events-only recording to /srv/frigate — telep-mainframe
  • Exposed the Frigate UI at https://telep-mainframe.taild4189d.ts.net via Tailscale Serve (real LE cert, tailnet-only, no LAN exposure) — Access
  • Built the HDMI camera wall: birdseye 3840x1080 → mpv straight to DRM/KMS with no X server; required nvidia-drm modeset=1 or the 3080 exposes zero display connectors — TV wall (HDMI)
  • Wired Telegram alerting (bot “Dezsi az őr” → supergroup) on person + car via frigate-notify polling the HTTP API every 15s — Alerting
  • Big gotcha: the dual-lens Tapo’s /cam1/ and /cam2/ RTSP paths are ALIASES FOR THE SAME LENS (proved by PSNR ≈36 vs ≈3.9); caused duplicate events + duplicate alerts and an invisible PTZ lens. Correct map: fixed=/stream1, PTZ=/stream6 — are the SAME lens
  • Recorded eight more gotchas: Secure Boot rejecting the DKMS-signed NVIDIA module, TensorRT detector dead on amd64 in 0.17 (use onnx), Frigate’s YOLOv9 export broken by torch ≥2.6, Tapo silently truncating long passwords, Frigate’s DB wiped unless the whole /config dir is mounted, birdseye not drawing camera names, birdseye grid geometry, iOS Private Relay breaking MagicDNS — Gotchas
  • Added the isolated telep-cc camera VLAN to the router doc (192.168.30.0/24, hidden SSID, WPA2 forced by Tapo, no cams → wan, NTP DNAT’d back) and recorded the router’s outstanding security gaps — Camera VLAN (telep-cc), Security Gaps

2026-07-13

  • Initialized the homelab project (physical/on-prem infra, separate from cloud-only levandor-infra) — homelab
  • Surveyed the OpenWrt router end-to-end: RT-AX89X on OpenWrt 24.10.0, LAN/WAN/firewall/DHCP, WPA3-SAE WiFi, Tailscale 1.80.3, 174 stock packages — telep-router
  • Fixed hostname drift: device still identified as OpenWrt (kernel, uname -n, UCI) while telep-router was only a Tailscale MagicDNS name. Set uci system.@system[0].hostname='telep-router', committed, reloaded — Changes Made (2026-07-13)
  • Recorded three gotchas: wifi-ifaces declaring network='lan wan' (inert but a footgun), 8-char WiFi PSK, and double-NAT behind 192.168.254.1 — Gotchas

Graph View