Full build of telep-mainframe — the bare-metal Debian 13 box that runs the Frigate NVR for the Telephely site: GPU object detection, event recording, an HDMI camera wall, and Telegram alerting. Cameras live on an isolated VLAN behind telep-router.
For Agents
Access:ssh levander@telep-mainframe (tailnet MagicDNS) or 192.168.1.123 on the LAN. Passwordless sudo.
Stack lives at:~/nvr on the host (docker-compose.yml + frigate/config.yml). A copy is at ~/telep/nvr/ on the Mac.
UI:https://telep-mainframe.taild4189d.ts.net — tailnet only, real Let’s Encrypt cert via Tailscale Serve. Not exposed on the LAN.Detector is onnx, NOT tensorrt — see 2. TensorRT detector is dead on amd64 before touching detector config.
The dual-lens camera’s /cam1/ and /cam2/ paths are the same lens — see 4. The Tapo’s /cam1/ and /cam2/ are the SAME lens. Use /stream1 and /stream6.
Host
This is bare metal, not a VM
systemd-detect-virt → none. There is no hypervisor to fall back on: a bad kernel/driver/modeset change means a physical trip to the site. Treat GPU and display changes accordingly.
Caddy :443 boot-race — a reboot can silently kill ALL domain access (found + fixed 2026-08-17)
caddy.service starts at boot before the LAN IP 192.168.1.123 is assigned to enp5s0, fails to bind :443 (bind: cannot assign requested address), and does not retry → every *.telep.lan service AND both levandor.io portals (admin/id) go down by-domain, silently. It sat failed for ~1.6 days after the 2026-08-15 YubiKey reboots before anyone noticed. Fix applied:net.ipv4.ip_nonlocal_bind=1 in /etc/sysctl.d/99-caddy-nonlocal-bind.conf (the real fix) + a Restart=on-failure / After=network-online.target drop-in at /etc/systemd/system/caddy.service.d/resilience.conf. If it recurs: systemctl status caddy → systemctl start caddy (and check oauth2-proxy is up, else admin.levandor.io 502s). Full incident Caddy boot-race outage (2026-08-17) · runbook caddy-boot-race.
Mains power is unstable and is CONFIRMED damaging the NVMe (as of 2026-08-05)
This box hard-cuts on mains power loss (NVMe SMART: 240 unsafe_shutdowns, 1617 media_errors). A UPS (pure-sine + AVR + NUT auto-shutdown) is arriving ~2026-08-07 — until then, do not run heavy/write-heavy builds (each cut mid-write adds media errors). Full root cause + resume plan: 2026-08-05-power-root-cause-nvme-damage-ups-kb-handover.
Disk unlock CHANGED 2026-08-15 — now YubiKey-gated, TPM/clevis REMOVED
Root LUKS no longer auto-unlocks via clevis/TPM2. It now unlocks via a YubiKey HMAC-SHA1
challenge-response initramfs keyscript — a possession-gated model: the YubiKey lives plugged
into the box (auto-unlocks unattended, survives mains cuts), but pull it → a udev rule syncs +
powers the box off, and booting without it → passphrase prompt. Root keyslots are now
0 = passphrase (break-glass), 2 = YubiKey; no TPM token, no slot 1. ⚠️ Never run ykman config … while the live-lock is armed (re-enumeration = poweroff) — disarm 99-yk-lock.rules
first. Full design + recovery: 2026-08-15-yubikey-gated-luks-fde · yubikey-luks-lockout.
The clevis+TPM2 description below is HISTORICAL (superseded note).
Verified 2026-07-31 — kernel/firmware updates and reboots are SAFE (unattended)
This bare-metal box (a physical trip to fix) reboots cleanly with no console interaction required:
Root is LUKS (nvme1n1p3, UUID a3a8e37d-…), auto-unlocked at boot. ⚠️ As of 2026-08-15 this is via a YubiKey keyscript, NOT clevis/TPM2 — see the banner above. (Historical: it was a clevis + TPM2 binding {"hash":"sha256","key":"ecc"} with NO pcr_ids, now unbound.) The unlock still happens early in the initramfs with no prompt while the YubiKey is present.
No dropbear remote-unlock fallback is installed — the keyscript (YubiKey present) or the console passphrase are the only unlock paths; a box stuck at the prompt is reachable only at the physical keyboard.
NVIDIA is DKMS (nvidia-current 550.163.01) — the module rebuilds automatically for each new kernel (rebuilt for 6.12.100). Still gated by Secure Boot being off (see 1. Secure Boot silently blocked the NVIDIA module).
The 2026-07-31 reboot (kernel 6.12.95→6.12.100) brought all services back cleanly: camwall, knowledgebase, frigate, kb-qdrant.
Headless hardening
The Debian installer ticks the desktop task by default — GNOME came along for the ride and had to be pushed out of the boot path. A machine whose entire job is to never stop recording must also never sleep:
getty@tty1 is disabled — mpv owns tty1 for the TV wall (HDMI). tty2–tty6 and SSH remain available.
Passwordless sudo — accepted tradeoff
/etc/sudoers.d/levander grants NOPASSWD. Convenient for automation, but it means any shell as levander is instantly root. Flagged deliberately; can be removed if the box’s exposure changes.
Architecture
Data flow
graph LR
CAM["<b>Tapo TC47</b> (dual-lens)<br/>192.168.30.119<br/><i>telep-cc VLAN, no internet</i>"]
G2["<b>go2rtc</b><br/>restreams each RTSP once"]
DET["<b>Detect</b><br/>720p sub @ 5fps"]
REC["<b>Record</b><br/>2304x1296 main"]
ONNX["<b>ONNX detector</b><br/>YOLOv9-s 640x640<br/>RTX 3080 · ~12ms"]
DB["<b>/srv/frigate</b><br/>events, 14d"]
BE["<b>Birdseye</b><br/>3840x1080 RTSP"]
MPV["<b>camwall.service</b><br/>mpv → DRM/KMS → HDMI"]
FN["<b>frigate-notify</b><br/>polls HTTP API /15s"]
TG["<b>Telegram</b><br/>Dezsi az őr"]
TS["<b>Tailscale Serve</b><br/>:443 → 127.0.0.1:8971"]
CAM -->|RTSP| G2
G2 --> DET
G2 --> REC
DET --> ONNX
ONNX --> DB
REC --> DB
G2 --> BE --> MPV
DB --> FN --> TG
DB --- TS
style CAM fill:#3d2020,stroke:#c05050,color:#fff
style ONNX fill:#264653,stroke:#2a9d8f,color:#fff
style G2 fill:#2d2d2d,stroke:#888,color:#fff
style DET fill:#2d2d2d,stroke:#888,color:#fff
style REC fill:#2d2d2d,stroke:#888,color:#fff
style DB fill:#264653,stroke:#2a9d8f,color:#fff
style BE fill:#2d2d2d,stroke:#888,color:#fff
style MPV fill:#2d2d2d,stroke:#888,color:#fff
style FN fill:#2d2d2d,stroke:#888,color:#fff
style TG fill:#264653,stroke:#2a9d8f,color:#fff
style TS fill:#264653,stroke:#2a9d8f,color:#fff
The stack (~/nvr)
Two containers, no MQTT broker:
Service
Image
Version
frigate
ghcr.io/blakeblackshear/frigate:stable-tensorrt
Frigate 0.17.2
frigate-notify
ghcr.io/0x2142/frigate-notify:latest
v0.5.4
frigate-notifypolls the Frigate HTTP API every 15s instead of subscribing to MQTT — one less moving part to run and monitor.
The -tensorrt image is still the right image
Despite 2. TensorRT detector is dead on amd64, stable-tensorrt remains correct for NVIDIA — it is simply the CUDA build of Frigate. Only the detector type must be onnx.
Detection
Setting
Value
Detector
onnx
Model
YOLOv9-s @ 640x640
Path
/config/model_cache/yolo.onnx
Labelmap
/labelmap/coco-80.txt
input_tensor
nchw
input_dtype
float
Inference
~12 ms
GPU load
~6%
Tracked objects: person, car.
Video
go2rtc restreams each RTSP stream exactly once — the Tapo caps concurrent RTSP clients, so detect + record + birdseye must all feed from the restream, not from the camera directly.
Detect on the 720p sub stream @ 5 fps; record the 2304x1296 main.
ffmpeg hwaccel_args: preset-nvidia (NVDEC).
Recording
record.retain.days: 0 — events only, no 24/7 continuous recording.
Alerts / detections retained 14 days.
Storage: /srv/frigate.
Recording posture changed — verified 2026-07-31: main → record is now continuous 24/7 on all 4 streams
The record.retain.days: 0 above no longer matches config.yml. As verified 2026-07-31, each of the 4 lens-streams runs sub → detect + main → record full-HD continuous 24/7. That sustained HD-record bandwidth is what saturates the 2.4 GHz camera WiFi (99% busy) and bufferbloats the camera wall — the root cause of the camwall lag: 2026-07-31-camwall-lag-24ghz-wifi-bufferbloat. Reconcile this section when the intended posture is confirmed.
Networking
All ports bound to 127.0.0.1 only — 8971 (UI), 5000 (internal API), 8554 (RTSP/birdseye). Nothing listens on the LAN.
Access
Tailscale Serve terminates TLS with a real Let’s Encrypt certificate:
Requires HTTPS certs enabled in the tailnet admin console.
Persists across reboots.
Frigate’s own admin login still applies on top of the tailnet ACL — two layers.
iOS: iCloud Private Relay breaks MagicDNS in Safari
*.ts.net resolves to “website not found” on iPhone/iPad with Private Relay (or Limit IP Address Tracking) enabled. Turn it off for the tailnet to resolve. This is a client-side setting, nothing on the server can fix it.
TV wall (HDMI)
A wall-mounted TV shows a live multi-camera view, rendered straight to DRM/KMS with no X server at all.
Once a second camera was added, Frigate birdseye proved unreliable as a fixed wall source (it flip-flops between showing 2 and 4 cameras). camwall-mpv.sh now composites the 4 go2rtc substreams directly into a grouped 2×2 (DÉL left, ÉSZAK right) via an mpv --vf lavfi filtergraph. The birdseye description below is retained for the RTSP-pipeline isolation method it references. Full detail: 2026-07-28-camwall-4-substream-composite.
Wall now runs under Xorg :0, NOT DRM/KMS — this section is stale (2026-08-04)
As found while adding the AirPlay receiver, the camera wall runs under an Xorg :0 server, and the X server was split out of camwall.service into its own persistent camwall-x.service so stopping the grid no longer kills X. camwall.service is now just a client of :0. The DRM/KMS mpv description below predates the go2rtc/X wall rebuild — trust the X-based architecture in 2026-08-04-telep-mainframe-airplay-receiver-uxplay for anything touching the display or casting.
Frigate birdseye: enabled, restream: true, mode: continuous, 3840x1080, quality 8 → published at rtsp://127.0.0.1:8554/birdseye.
camwall.service (systemd) runs /usr/local/bin/camwall.sh, which is mpv:
Camera name labels are burned in with an ffmpeg drawtext filter — hence nvdec-copy rather than nvdec: a software filter needs the frames back in system memory.
The 3080 exposes NO display connectors until DRM modesetting is on
Out of the box the GPU showed zero connectors — no HDMI, nothing to render to. Fix:
/etc/modprobe.d/nvidia-drm.conf:
options nvidia-drm modeset=1
then update-initramfs -u. It was applied live without a reboot by stopping frigate and reloading nvidia_drm.
A second Tapo TC47 (ÉSZAK, 192.168.30.139) was added alongside the original (DÉL, 192.168.30.119). Each is dual-lens (Fix + PTZ), so Frigate now has 4 cameras / 4 go2rtc substreams:
lan → cams allowed (this is how Frigate pulls RTSP)
Clock
NTP DNAT’d back to the router so the offline camera does not drift
Hidden-SSID rejoin was actually tested: the camera was deauthed, and it re-associated and kept its lease. Tapo handles hidden SSIDs fine on this firmware.
Consequence: the Tapo app can no longer reach the camera
No cloud (no internet) and the phone sits on a different subnet with client isolation on. Changing any camera setting now requires temporarily reopening the VLAN. Budget for this before planning camera-side changes.
RTSP stream map
Lens
Main
Sub
Fixed
/stream1 (2304x1296)
/stream2 (1280x720)
PTZ
/stream6
/stream7
Alerting
Telegram bot “Dezsi az őr” (@tlphlyortbot) → supergroup Telephely biztonsági riasztások, chat_id -1004475187307.
Credentials: ~/telep/telegram-bot.env on the Mac (mode 600).
frigate-notify alerts on person and car.
Alerts are wanted only on DÉL / South (192.168.30.119 = telep_cam1/telep_cam2); ÉSZAK / North (192.168.30.139 = telep_cam3/telep_cam4) is excluded on purpose.
Alerts were DEAD ~17 days (2026-07-29 → 2026-08-15) — Tailscale MagicDNS broke Docker container external DNS
Alerts silently stopped at exactly 2026-07-29 13:01:15 and nobody noticed (there is no failure alarm). The logs flooded with Already notified on this zone (empty zones=), which looks like a dedup/zones bug but is NOT — frigate-notify caches an event (otter cache, 1 h TTL, keyed by event ID) before the Telegram send, so a failed send stays cached and every 15 s poll re-logs “Already notified”.
Root cause: the host /etc/resolv.conf became nameserver 100.100.100.100 (Tailscale MagicDNS) on 2026-07-29. Docker’s embedded DNS (127.0.0.11) forwards container external lookups to that upstream, and MagicDNS misbehaves for containers → frigate-notify could not resolve api.telegram.org (buried WRN: ... server misbehaving). Bot/token/snapshots/config were all fine.
Fix: pinned dns: [192.168.1.1, 1.1.1.1] on the frigate-notify service in ~/nvr/docker-compose.yml (backup docker-compose.yml.bak-dns), docker compose up -d --force-recreate frigate-notify. Alerts fired instantly. Any container needing external DNS on this host has the same trap. Full write-up + runbook: 2026-08-15-camera-alerts-dead-tailscale-magicdns-docker-dns, camera-alerts-not-firing.
Diagnostic — "Already notified" flood with zero "Alert sent" = a buried delivery error
docker logs --tail 4000 frigate-notify | grep WRN → look for Unable to send alert / server misbehaving. It’s a DNS/delivery failure, not dedup. Don’t touch zones config to fix a silent-alerts outage.
Per-camera alert exclusion (2026-07-28)
telep_cam3 (ÉSZAK – Fix, aimed indoors) is excluded from frigate-notify alerts via frigate.cameras.exclude (nested under the frigate: block, by camera id — notalerts.cameras.block, not top-level cameras:). koanf silently ignores misplaced keys, so verify suppression against a real event (Event dropped - Camera Excluded), never against “Config validated!“. Full detail + re-enable steps: 2026-07-28-frigate-notify-camera-exclude.
Gotchas
The expensive ones, in the order they cost time.
1. Secure Boot silently blocked the NVIDIA module
DKMS built and signed nvidia.ko with a MOK that was never enrolled, so modprobe nvidia failed with:
Key was rejected by service
Fix chosen: disable Secure Boot in the BIOS. Enrolling the MOK is the “proper” fix, but MOK Manager throws up a blue enrollment screen that requires a physical monitor and keypress — a non-starter on a headless box. Use mokutil --test-key to confirm enrollment state before assuming a signing problem.
2. TensorRT detector is dead on amd64
On Frigate 0.17 the TensorRT detector errors out immediately:
TensorRT detector is no longer supported on amd64 system.
Please use ONNX detector instead.
The -tensorrtimage is still correct (it is the CUDA build) — only the detector must be onnx. Easy to conflate the two and chase the wrong thing.
3. Frigate's YOLOv9 export recipe is broken by PyTorch ≥ 2.6
Frigate’s export Dockerfile pins nothing, so it pulls torch 2.13. Torch 2.6 flipped torch.load to weights_only=True by default, and yolov9’s export.py never passes weights_only=False:
_pickle.UnpicklingError: Weights only load failed
Fix — pin torch, after the requirements install:
RUN uv pip install --system torch==2.5.1 torchvision==0.20.1 \ --index-url https://download.pytorch.org/whl/cpu
4. The Tapo's /cam1/ and /cam2/ are the SAME lens
This was the big one. Probing suggested three working paths — /cam1/stream1, /cam2/stream1 and /stream1 all returned 200. But a PSNR comparison proved /cam1/* and /cam2/* are the same fixed lens (PSNR ≈ 36 → identical scene), while /stream6 is a genuinely different scene (PSNR ≈ 3.9).
Symptoms: the UI showed the same feed twice; every detection produced two identical events and two Telegram alerts; and the PTZ lens moved but was never actually visible anywhere.
Correct mapping: fixed = /stream1 + /stream2; PTZ = /stream6 + /stream7. An HTTP 200 on an RTSP path proves nothing about which sensor is behind it — compare frames.
5. Tapo silently truncates long camera-account passwords
A 28-character password was accepted by the Tapo app but never authenticated (401 forever). A 14-character alphanumeric password worked.
Also worth knowing:
The camera only offers Basic auth — WWW-Authenticate: Basic realm="TP-Link IP-Camera". Not digest.
Reboot the camera after changing the camera account, or RTSP keeps rejecting the new credentials.
6. Frigate keeps its database at /config/frigate.db
Bind-mounting only config.yml into /config leaves the event DB container-local — it is wiped on every docker compose up --force-recreate. Mount the whole directory:
volumes: - ./frigate:/config
7. Birdseye does not draw camera names
There is no putText in birdseye.py. Labels on the TV must be burned in at the player (mpv drawtext). Do not go looking for a Frigate config option — there isn’t one.
8. Birdseye grid geometry is driven by the canvas aspect
On a 16:9 canvas with 2 cameras, birdseye builds a 2x2 grid with an empty dark bottom row. Setting the canvas to 3840x1080 (32:9) forces a single row of two.
Note the inherent limit: two 16:9 cameras can never fill a 16:9 screen without cropping. Some letterboxing is unavoidable.
ANPR / license plates. Frigate 0.17 supports LPR natively (lpr: enabled, device: GPU, known_plates with wildcards/regex). But the current optics cannot resolve a plate — LPR needs roughly 100 px across the plate, and the wide lens at night gives about 20 px. Requires a dedicated entrance camera with illumination; no amount of config fixes this.
Define zones. Nothing is zoned yet — this is what makes frigate-notify’s dedup precise.
End-to-end test: person → Telegram, with a real human in frame. Not yet done.
tailscale serve --https=8450 → 127.0.0.1:3010; Caddy-independent path to the “where is what” board
The Homepage “where is what” status dashboard also lives on this box at /home/levander/homepage/ (Docker, network_mode: host, binds 127.0.0.1:3010) and is served on the LAN at http://home.telep.lan via the :80 Caddy. See 2026-08-17-home-dashboard.
NVMe device nodes on this box are not stable across reboots
As of 2026-07-21 the 1.8T Windows disk is nvme0n1 (nvme0n1p2 = NTFS) and the 3.6T LUKS+LVM OS disk is nvme1n1 (p1 /boot/efi, p2 /boot ext4, p3 crypto_LUKS) — the reverse of the earlier enumeration. Always lsblk -f before mounting anything; use UUID= in fstab/systemd units. Details and the mount procedure: The Windows drive.
Other drives: sda (447.1G) is essentially unused (one 16M partition, no filesystem); sdb (115.5G) is a Hiren’s BootCD PE rescue USB stick (UDF, label HBCD_PE_x64).
GPU OCR venv (~/ocr/venv)
A reusable marker-pdf install lives at ~/ocr/venv for turning scanned PDFs into markdown on the RTX 3080 (564 pages in ~22 min, ~3.5 GB VRAM, no impact on Frigate).
Its torch was hand-swapped to a cu126 build — do not "fix" it
The default marker-pdf install pulls a cu130 torch that this box’s driver 550.163.01 cannot initialise, so it silently runs on CPU. And do not upgrade the NVIDIA driver to chase CUDA 13 — driver 550 is load-bearing for Frigate detection and the camera wall. Full procedure: Running it on a CUDA GPU box (telep-mainframe).
Also running (no served port): intruder-alarm.service — presence-based auto-arming alarm that watches Frigate detections, plays a TV siren, and escalates intruders to Telegram. See 2026-07-17-intruder-alarm.
Related
2026-08-04-telep-mainframe-airplay-receiver-uxplay — this box is now an AirPlay receiver (telep-tv, uxplay 1.71); the enabling change split Xorg :0 out of camwall.service into a persistent camwall-x.service so casting can pause/restore the camera grid. Software avdec_h264 (no HW decode), xvimagesink+xdotool 4K resize, ALSA plughw:1,3 audio
2026-07-31-camwall-lag-24ghz-wifi-bufferbloat — camwall lag ROOT CAUSE: the 4 continuous HD record streams saturate the 2.4 GHz camera WiFi (99% busy → bufferbloat); an RF/bandwidth problem upstream of mpv/go2rtc
2026-07-17-intruder-alarm — auto-arming alarm on this box; watches Frigate, sirens the TV, escalates to Telegram
2026-07-31-telep-mainframe-ax210-ble-scan-broken — onboard AX210 saw zero BLE devices → root cause was a MISSING ANTENNA (WiFi+BT share the M.2 u.FL connector); blocked the thermoprint appliance. Lesson: check physical RF before firmware/kernel/reboot. Also records the reboot-safety facts (LUKS clevis+TPM2 no-PCR auto-unlock, no dropbear, NVIDIA DKMS)
telep-router — the gateway, the telep-cc camera VLAN, and the tailnet path in
2026-08-03-telep-router-factory-reset-recovery — this box was the SSH jump host for rebuilding the factory-reset router; its Frigate feeds + both cameras were re-verified after. Note the router’s tailnet IP changed to 100.69.112.32
filestash — Windows drive file browser sharing this host’s tailscale serve pattern