Levandor

Camwall black after power outage — Xorg HDMI mode fell back to 640x480, self-heal added

5 min read

After a power outage the TV camwall on telep-mainframe “didn’t really come back” — the TV was black/dead even though the host itself never rebooted (it’s on the UPS, uptime 3d17h). Root cause: after the power/HDMI renegotiation Xorg fell back to 640x480 with NO mode set on the HDMI output, so the four 1920x1080 mpv panes tiled across a 3840x2160 screen couldn’t map → all black. Nothing auto-corrected it because both existing guards were blind to a wrong resolution (the HDMI stayed “connected”, and the stall watchdog skips null time-pos). Fixed with a self-healing xrandr on every camwall start plus a resolution check in the watchdog; both verified live via fault injection.

For Agents — Quick Reference

Symptom class: camwall black/dead after a power blip, host did NOT reboot (UPS held it up; power event only restarted services: Frigate, camwall-x/Xorg, camwall). The tell: DISPLAY=:0 xrandr shows Screen 0: current 640 x 480 and HDMI-1 connected with NO active geometry, while the preferred 3840x2160@60 is listed as available. DRM connector = card1-HDMI-A-3. Manual recovery: DISPLAY=:0 xrandr --output HDMI-1 --primary --mode 3840x2160 --pos 0x0 --fb 3840x2160 then systemctl restart camwall. Now self-heals automatically via two changes below (worst case ~10–20s of fallback before the watchdog re-rolls, POLL=10s). Screenshot the wall: DISPLAY=:0 /usr/bin/ffmpeg -f x11grab -video_size 3840x2160 -i :0 -frames:v 1 out.png — host has NO ffprobe; host ffmpeg is /usr/bin/ffmpeg (NOT /usr/lib/ffmpeg/7.0/..., which is the Frigate container’s). Backups: camwall-mpv.sh.bak-mode-1786483068, camwall-watchdog.py.bak-mode-1786483068.

Incident

  • ~23:00 CEST, 2026-08-11, after a power outage: the TV camwall was black/dead.
  • The host did NOT reboot — it’s on the UPS, uptime 3d17h. The power blip only triggered service restarts: Frigate, camwall-x/Xorg, and camwall.
  • So this was not a boot-recovery failure — it was a display-mode failure that survived every automatic re-roll.

Root cause — Xorg fell back to 640x480 on the HDMI output

After the power/HDMI renegotiation, Xorg brought the HDMI-1 output up at 640x480 with no mode actually set, even though the panel’s preferred 3840x2160@60 was available in the mode list. The camwall tiles four 1920x1080 mpv panes across a 3840x2160 screen; on a 640x480 root the panes couldn’t map and all four went black (mpv time-pos = null on every pane).

Diagnosis clincher — it was a display-mode problem, not a stream or GPU problem:

  • xrandr: Screen 0: current 640 x 480, HDMI-1 connected with no active geometry, 3840x2160@60 present-but-unused.
  • go2rtc streams were healthy: ffprobe on cam1_sub = h264 1280x720, all producers connected.
  • GPU (RTX 3080) was fine.

Why nothing auto-corrected it

Both existing guards were blind to a wrong-resolution HDMI:

  • The camwall-watchdog only watched HDMI connect/disconnect — the output stayed “connected” the whole time, just at the wrong mode, so its HDMI logic never tripped.
  • Its stall path skips null time-pos (treats an unreadable pane as “not a stall”, not a freeze), so the four black panes never counted as stalls either. This is the same null-handling described in 2026-08-11-camwall-freeze-and-cam-reboot-fixes.
  • Every re-roll (including camwall-frigate-watch firing when Frigate restarted) came back up on the same 640x480 screen → black again.

Immediate recovery

DISPLAY=:0 xrandr --output HDMI-1 --primary --mode 3840x2160 --pos 0x0 --fb 3840x2160
systemctl restart camwall

Screen returned to 3840x2160, all 4 panes live — verified by an x11grab screenshot showing DÉL/ÉSZAK fix+PTZ live video.

Durable fix (deployed + verified via fault injection)

Two layers: self-heal on every start, and a watchdog that catches a mid-run fallback.

Change 1 — /usr/local/bin/camwall-mpv.sh self-heals the mode on every start/re-roll

Added, near the top (after the xset lines, before launching the panes):

xrandr --output HDMI-1 --primary --mode 3840x2160 --pos 0x0 --fb 3840x2160 2>/dev/null || true

So every camwall start/re-roll forces the correct mode first. Backup camwall-mpv.sh.bak-mode-1786483068.

  • Verified: forcing xrandr --output HDMI-1 --mode 640x480 then a systemctl restart camwall restored 4K.

Change 2 — /usr/local/bin/camwall-watchdog.py resolution check in the poll loop

Added a resolution check to the existing poll loop:

  • Runs xrandr with env DISPLAY=:0 (root under systemd reads the local Xorg directly — no XAUTHORITY needed).
  • Parses Screen 0: ... current W x H.
  • If not 3840x2160, calls the existing reroll("resolution fallback WxH").
  • Preserves the existing all-4-pane stall detection + HDMI connect/disconnect logic from 2026-08-11-camwall-freeze-and-cam-reboot-fixes.

Backup camwall-watchdog.py.bak-mode-1786483068.

  • Verified (no manual action): forcing xrandr --output HDMI-1 --mode 640x480 auto-restored to 4K within ~22s via the watchdog. Journal: 
    camwall-watchdog.py: rerolling camwall: resolution fallback 640x480
  • POLL=10s → worst case ~10–20s of fallback before auto-heal.

Context for future agents

  • No ffprobe on the host. Host ffmpeg is /usr/bin/ffmpeg; /usr/lib/ffmpeg/7.0/... belongs to the Frigate container. Screenshot the wall with DISPLAY=:0 /usr/bin/ffmpeg -f x11grab -video_size 3840x2160 -i :0 -frames:v 1 out.png.
  • The watchdog skips null mpv time-pos (unreadable ≠ stall) — that’s precisely why black panes from a mode-fallback are NOT caught by the stall path. The new resolution check is what covers this failure mode.
  • DRM connector for this output is card1-HDMI-A-3; the xrandr name is HDMI-1.

Graph View

Backlinks