Levandor

Frigate per-camera FPS watchdog (frozen-feed + detection-jam auto-restart)

3 min read

A new systemd-managed watchdog on telep-mainframe that polls Frigate’s /api/stats and auto-restarts the frigate docker container when it detects either a frozen camera feed or a stalled detection pipeline, with a circuit breaker so a genuinely-offline camera can’t cause restart thrash.

For Agents

Script: /usr/local/bin/frigate-fps-watchdog.py Unit: /etc/systemd/system/frigate-fps-watchdog.service — active, enabled, User=root, Restart=always. Polls: http://127.0.0.1:5000/api/stats every 20 s. Two failure modes caught:

  1. Frozen feed — camera_fps < 1.5 (normal ~5) for a single camera.
  2. Detection jam — process_fps < 1.0 while camera_fps is healthy. Action: docker restart frigate. Telegram: alerts via /etc/nut/telegram.env (same creds as frigate-notify / NUT). All tunables are constants at the top of the script.

Why this exists

Today (2026-08-08) Frigate jammed twice, each needing a manual sudo docker restart frigate:

  1. A whole-pipeline detection jam (the CPU-starvation failure mode documented in 2026-08-06-frigate-detect-record-jam-cpu-starvation — process_fps collapses while camera_fps looks normal).
  2. A frozen single-camera feed — cam3 dropped to camera_fps = 0.3 (a wedged/stalled input, distinct from detection stalling).

The existing camwall watchdogs (camwall-watchdog, camwall-frigate-watch) watch the display/grid side, not per-camera Frigate fps — so neither of today’s failures would have been caught automatically. This watchdog closes that gap by watching the numbers Frigate itself reports.

Detection logic

Polls GET http://127.0.0.1:5000/api/stats every 20 s and evaluates two independent conditions:

Failure modeConditionNormal value
Frozen feedcamera_fps < 1.5 for a camera~5
Detection jamprocess_fps < 1.0 while camera_fps healthyprocess_fps tracks camera_fps

Either condition trips the watchdog. Both resolve to the same remedy: docker restart frigate.

Bulletproofing (why it won’t thrash)

The hard design constraint: a camera that is genuinely offline (unplugged, dead WiFi, rebooting) will show low fps indefinitely — a naive watchdog would restart Frigate in an endless loop. Three guards prevent that:

  1. Debounce — a condition must persist for 3 consecutive polls (~60 s) before any restart. Rides out transient dips.
  2. Post-restart cooldown — 120 s of silence after a restart before the watchdog will act again (Frigate needs time to warm up; restarting mid-warmup would false-positive).
  3. Circuit breaker — max 3 restarts per hour. The 4th restart within the hour is suppressed → instead the watchdog sends a Telegram alert only and re-alerts every 30 min while the condition persists. This converts “restart thrash” into “human, come look” for a camera that a restart can’t fix.

A restart doesn't fix everything

The circuit breaker exists precisely because some failures are NOT restart-fixable. A wedged camera-side RTSP session (see 2026-08-03-cam-stall-recovery-and-casino-alert) survives a Frigate restart and needs an ONVIF camera reboot instead. When the breaker trips, that’s the signal to check whether the camera itself needs rebooting.

Tuning

All thresholds live as constants at the top of /usr/local/bin/frigate-fps-watchdog.py:

  • poll interval (20 s), debounce count (3), post-restart cooldown (120 s)
  • camera_fps floor (1.5), process_fps floor (1.0)
  • circuit-breaker window (1 h), max restarts (3), re-alert interval (30 min)

Relationship to the manual runbook

This automates the two manual fixes previously in the runbook:

It does not replace the ONVIF-reboot recovery for a wedged camera — that remains manual (and is exactly what a tripped circuit breaker points you toward).

Graph View

Backlinks