Levandor

Camwall per-pane freeze detection + working ONVIF camera reboot with post-reboot clock re-sync

6 min read

Cameras on the TV camwall (telep-mainframe) looked “stuck for a long time” — a quadrant sat frozen on a static image with a stopped OSD clock while Frigate ingest stayed healthy (camera_fps ~5). Three root causes found and fixed: (1) the camwall stall-watchdog only watched one of the four mpv panes, so a stalled cam1/cam3/cam4 pane was never detected or re-rolled; (2) the pytapo camera-reboot tooling had silently died on Tapo KLAP auth breakage; (3) a reboot wipes the RTC-less Tapo clocks, leaving ~40 min of wrong OSD time until the hourly ONVIF push self-heals. All three applied and verified end-to-end. Closes the open “watchdog covers only the IPC-socket-owning pane” follow-up flagged in 2026-07-28-camwall-4-substream-composite.

For Agents — Quick Reference

Camwall: /usr/local/bin/camwall-mpv.sh (camwall.service) — 4 tiled single-input mpv panes, each a go2rtc substream at rtsp://127.0.0.1:8554/: cam1_sub (DÉL-Fix, TL), cam2_sub (DÉL-PTZ, BL), cam3_sub (ÉSZAK-Fix, TR), cam4_sub (ÉSZAK-PTZ, BR). Per-pane IPC sockets (NEW): /run/camwall-cam1.sock … /run/camwall-cam4.sock (was: only /run/camwall.sock on the cam2 pane). Stall watchdog: /usr/local/bin/camwall-watchdog.py (camwall-watchdog.service) — now polls time-pos on ALL FOUR sockets, per-socket stall counters, re-rolls the whole wall (systemctl restart camwall) if ANY pane hits STALL_LIMIT. HDMI-reconnect logic preserved. Frigate re-roll watch: camwall-frigate-watch.service — re-rolls camwall when Frigate restarts (unchanged). Camera reboot (WORKING): ~/tapo-ctl/reboot-cams-onvif.py — ONVIF SystemReboot() on port 2020 using Frigate RTSP creds, waits for ping-back, then re-syncs OSD clock via set-cam-time.py. Default reboots both cams (192.168.30.119, 192.168.30.139). Clock re-sync: ~/tapo-ctl/set-cam-time.py (creds from Frigate container config + docker env) + hourly cam-timesync.timer (see 2026-08-08-camera-osd-clock-drift-onvif-timesync). DEAD: ~/tapo-ctl/reboot-cam.py / reboot-onvif.py (pytapo, hardcoded nvr42vhy1 creds — Invalid authentication data on both cams). Left in place but non-functional. Backups: camwall-mpv.sh.bak-1786433100, camwall-watchdog.py.bak-1786433100.

Symptom vs. why the existing watchdogs missed it

The failure looked like “stuck cams” but every automatic guard reported healthy:

  • A single quadrant showed a static image with a frozen OSD clock (stalled stream — no new frames arriving), for a long time.
  • Frigate ingest was fine: camera_fps ~5, so the Frigate per-camera FPS watchdog saw nothing wrong (it watches Frigate’s /api/stats, not the display panes).
  • The camwall stall-watchdog existed but was blind to three of the four panes (see Root cause 1).

This is a display-side stall: the go2rtc substream to that one mpv pane stopped delivering frames and mpv held the last frame, while Frigate’s own ingest of the same camera kept running. Nothing tied the two together, so the quadrant stayed frozen.

Root cause 1 — camwall panes had no per-pane freeze detection (the real “stuck” cause)

The camwall runs four independent mpv processes (one per quadrant, from the tiled low-latency rearchitecture in Latency fix (2026-07-29) — 4 tiled single-input mpv instances). Previously only the cam2 pane exposed --input-ipc-server=/run/camwall.sock. camwall-watchdog.py checked mpv time-pos on that one socket.

A stalled cam1/cam3/cam4 pane was undetectable

If the cam1, cam3, or cam4 pane stalled (stream stops → static image, frozen OSD clock), nothing detected it and nothing re-rolled — that quadrant sat frozen indefinitely while Frigate ingest stayed healthy. Only a stall of the single socket-owning cam2 pane would ever trip a re-roll. This is exactly the “watchdog freeze-detection now only covers the one IPC-socket-owning pane, not all 4 — security-relevant” regression that was left open in 2026-07-28-camwall-4-substream-composite.

Fix — per-pane IPC sockets + all-four-pane polling

  • All four panes now get their own IPC socket: /run/camwall-cam{1,2,3,4}.sock (in /usr/local/bin/camwall-mpv.sh).
  • camwall-watchdog.py now polls time-pos on all four sockets, keeps a per-socket stall counter, and re-rolls the whole wall (systemctl restart camwall) if any pane reaches STALL_LIMIT.
  • HDMI-reconnect logic preserved unchanged.
  • Backups: camwall-mpv.sh.bak-1786433100, camwall-watchdog.py.bak-1786433100.

Verified end-to-end via stall simulation

Pausing the cam3 pane produced, ~33 s later:

camwall-watchdog.py: rerolling camwall: pane /run/camwall-cam3.sock stalled

…and the wall re-rolled (cam3 pane uptime reset to fresh). All four sockets respond to time-pos.

Scope: this catches STALLED-STREAM freezes, not repeated-frame freezes

The watchdog trips when time-pos stops advancing (no new frames). It does not catch a camera that keeps pushing a repeated frozen frame at full fps (there time-pos keeps advancing). The user confirmed the freezes on this wall are the static/stalled kind, so content-hash / framebuffer-staleness detection was not needed. If a repeated-frozen-frame failure ever shows up, that’s the direction to add.

Root cause 2 — camera reboot tooling was silently broken

The recovery for a wedged camera is an ONVIF reboot (established in 2026-08-03-cam-stall-recovery-and-casino-alert). The pytapo-based tool no longer works:

pytapo reboot path is dead — Invalid authentication data

~/tapo-ctl/reboot-cam.py (pytapo, hardcoded account nvr42vhy1 + password lifted from disable_alarm.py) now fails with Invalid authentication data on BOTH cameras (192.168.30.119, 192.168.30.139) — the classic Tapo firmware/KLAP auth breakage. pytapo is not a viable path here anymore.

Fix — ONVIF SystemReboot() with Frigate creds

The working path is ONVIF SystemReboot() on port 2020 using the Frigate RTSP creds — the same credential resolution set-cam-time.py uses (reads them from the Frigate container config + docker env), not the stale hardcoded nvr42vhy1 account.

New tool ~/tapo-ctl/reboot-cams-onvif.py:

  1. Reboots the given IPs (default: both cams) via ONVIF SystemReboot().
  2. Waits for each camera to ping back.
  3. Re-syncs the OSD clock by invoking set-cam-time.py (see Root cause 3).

Cleanup: the dead ~/tapo-ctl/reboot-cams.py (an earlier pytapo attempt) was removed. The user’s original hardcoded reboot-cam.py / reboot-onvif.py (.139 only, stale nvr42vhy1 creds) were left in place but are effectively dead.

Root cause 3 — wrong OSD time after a reboot

Tapo cams have no RTC, so a reboot wipes their clock. cam-timesync.timer re-pushes host time over ONVIF hourly (cam-timesync.service, “Push host time to Tapo cameras over ONVIF”), so it self-heals within an hour — but that leaves a ~40 min window of wrong OSD time after any reboot.

Fix: re-sync the clock immediately on reboot, don't wait for the hourly timer

reboot-cams-onvif.py calls set-cam-time.py as soon as the cameras come back up, closing the post-reboot gap. Both cams verified showing correct time after. Full background on the ONVIF time-push (pre-compensated for the Tapo TZ bug, DST-safe) is in 2026-08-08-camera-osd-clock-drift-onvif-timesync.

This session’s manual actions

  • Both cameras were manually rebooted via the new ONVIF tool — both healthy after (~5 fps).
  • The camwall was re-rolled.
  • camwall.service / /usr/local/bin/camwall-mpv.sh — the 4-pane tiled wall
  • camwall-watchdog.service / /usr/local/bin/camwall-watchdog.py — HDMI reconnect + now all-4-pane stall detection
  • camwall-frigate-watch.service — re-rolls camwall when Frigate restarts
  • cam-timesync.timer / cam-timesync.service — hourly ONVIF host-time push

Graph View

Backlinks