Levandor

Homelab alert-source inventory — six producers, four silent holes

18 min read

A source-level audit of every alert producer on telep-mainframe and telep-router, done while moving homelab alerting off Telegram and into Matrix (2026-09-07-session-handover §3). Six producers exist — two were already on Matrix, and the four remaining Telegram alerters were migrated later the same day (see ✅ The migration — what changed). The audit also found four alerting paths wired to nothing at all, a third copy of the leaked Telegram bot token, and one finding that reframes the whole migration: Telegram fails precisely during the events it exists to report.

✅ Sources 1–4 are MIGRATED — all four now send through the Matrix relay

This note previously read ”🚧 The migration of sources 1–4 is IN PROGRESS — not done”. All four host-side Telegram alerters were migrated on 2026-09-07, verified, and are running. Per-script rollback lines, verification and caveats are in ✅ The migration — what changed. Source 5 (casino-alert) remains unmigrated — infra-blocked, and source 6 (intruder-alarm) was never portable and has since been RETIRED (§6). 🟡 One thing must still NOT be recorded as done: the :8118 production path has never been exercised end to end. Everything was proven against a throwaway relay on :8119. The first real alert is the test.

For Agents

Hosts audited: telep-mainframe (192.168.1.123) and telep-router (192.168.1.1). Relay: /opt/matrix-relay/matrix-relay, binds 127.0.0.2:8118 + 172.18.0.1:8118 only. Target room: #telep-ertesitesek:chat.taild4189d.ts.net (encrypted). Test alerts belong in #relay-scratch:chat.taild4189d.ts.net. Telegram credentials live in THREE places — /etc/nut/telegram.env (0600), /home/levander/nvr/frigate-notify/config.yml, and hardcoded in /etc/casino-alert.sh on the router (0755, world-readable). No other notification stack exists on either host: no ntfy, gotify, pushover, discord, slack, healthchecks.io, Home Assistant, Uptime Kuma.

The inventory

#SourceHostChannelMigration verdict
—bambuddymainframe✅ Matrix (provider id=2)Done — id=1 Telegram disabled, not deleted
—frigate-notifymainframe✅ Matrix (webhook → 172.18.0.1:8118/frigate)Done
1frigate-fps-watchdogmainframe✅ Matrix ([fps-watchdog])✅ Migrated 2026-09-07 — fires most
2UPS outage handler (NUT)mainframe✅ Matrix ([ups])✅ Migrated 2026-09-07 — a correctness fix, see below
3top-kép remindersmainframe✅ Matrix ([top-kep])✅ Migrated 2026-09-07 — still a candidate for its own room
4frigate-viewer-alertmainframe✅ Matrix ([viewer-alert])✅ Migrated 2026-09-07 — highest volume
5casino-alertrouter📨 Telegram⛔ Infra-blocked — relay unreachable from the router
6intruder-alarmmainframe— (retired)✅ RETIRED 2026-09-07 — archived, not deleted; never portable (interactive bot)

1. frigate-fps-watchdog — the loudest one

/usr/local/bin/frigate-fps-watchdog.py + frigate-fps-watchdog.service, creds from /etc/nut/telegram.env. Full design in 2026-08-08-frigate-fps-watchdog.

This is the alerter that actually fires. 12+ genuine alerts on 2026-09-04 alone — camera restarts, a breaker trip, and the recovery notices afterwards. Whatever else moves, this one carries real operational signal.

2. UPS outage handler — the one that has never worked when it mattered

/etc/nut/nut-outage-handler.sh, invoked by upsmon as NOTIFYCMD, creds from /etc/nut/telegram.env. Background: 2026-08-05-power-root-cause-nvme-damage-ups-kb-handover, 2026-08-08-ups-shed-restore-hardening.

Real outages fired it on 2026-08-18, 08-19, 08-20 and 08-22. See the headline finding below for why almost none of them were delivered.

3. top-kép reminders — not security traffic

/home/levander/top_kep_remind.py driven by three systemd timers (daily 20:00, Sun 19:30, daily 21:00 monthly-gated). See 2026-08-10-top-kep-community-voting-system.

A family voting nudge, not an alarm. It has no business sharing a room with camera and power alerts — a candidate for its own Matrix room when it moves.

4. frigate-viewer-alert — highest volume, and unprovable from logs

/home/levander/frigate-viewer-alert/{watch,notify}.py, Restart=always. Design in 2026-07-23-frigate-viewer-alert.

Two corrections to the recorded behaviour, from reading the source:

  • It fires on any tailnet request to Frigate, not only /live/ — the note’s framing as “someone opened the camera UI” understates its trigger surface.
  • The camera field is optional in the message it builds.
  • Rate-gating is 300 s per source IP.

Its firing rate cannot be established from logs — by design

notify.send() prints only on failure, so a successful send leaves no trace anywhere. The evidence that it works is indirect: a stable PID for 4d22h, only 3 send failures in 60 days, and 98 tailnet X-Forwarded-For hits in the last 20k lines of the frigate log. Do not go looking for a success count — there isn’t one.

5. casino-alert (router) — infra-blocked, and double-running

/etc/casino-alert.sh + S95casino-alert (procd, respawn, enabled) on telep-router. Design in 2026-08-03-cam-stall-recovery-and-casino-alert.

⛔ It cannot be migrated as things stand

The relay binds 127.0.0.2:8118 and 172.18.0.1:8118 — a host loopback address and a docker-bridge address. Neither is reachable from the router. Migrating casino-alert requires an additional relay bind or a proxy first; it is an infrastructure blocker, not a scripting task.

🔴 casino-alert is running TWICE

Two PIDs, each with its own tail -F on the query log. Every match would therefore double-send. It has never been observed firing — but its cooldown state lives in /tmp/casino-alert-state and is wiped on every boot, so “never seen firing” is a weak observation window. Treat it as unproven, not dead.

6. intruder-alarm — ✅ RETIRED 2026-09-07

/home/levander/intruder-alarm.py, disabled and inactive at audit time. Design in 2026-07-17-intruder-alarm / 2026-07-17-intruder-alarm-design.

1175 lines of an interactive Telegram bot, not a one-way alerter: send_photo, edit_message_text, inline keyboards, and callback polling for MAC enrollment.

Matrix has no inline-keyboard equivalent

There is no direct port. Rebuilding the enrollment UX on Matrix would be a rewrite, not a migration — which is why the recommendation was to retire rather than port it.

✅ Retired the same day — archived, not deleted

Archive: /root/retired/2026-09-07-intruder-alarm/ holding intruder-alarm.service, intruder-alarm.py and a README. Removed from live paths: /etc/systemd/system/intruder-alarm.service and /home/levander/intruder-alarm.py are both confirmed gone; systemctl status returns “Unit intruder-alarm.service could not be found”; daemon-reload was run. Restore is one step: mv both files back and systemctl daemon-reload.

Safe because nothing depended on it — grep-verified. The only hit outside its own files was a comment in /home/levander/ruview/scripts/c6-presence-watcher.py (# so the operator never sees "intruder detected" framing), plus two coincidental word matches in unrelated ruview/examples/research-sota/ research files. Recorded so nobody re-derives it.

A side effect: one source of journal noise is gone

Its unit emitted Unknown key 'StartLimitIntervalSec' in section [Service] on every daemon-reload. That warning is now gone. ✅ There is no wider misconfiguration to chase — the three units that legitimately carry that key (camwall.service, camwall-x.service, nvidia-cdi-refresh.service) all have it correctly in [Unit].

🔴 Headline finding — Telegram fails exactly when it is needed

31 of 34 UPS alert sends FAILED — all of them inside the outage windows

Every failure clusters inside the 2026-08-07 (6 failures) and 2026-08-18 (25 failures) power events. The cause is structural, not flaky: the WAN was down because the power was out, so the alert could not leave the building. The only sends that succeeded landed after power and network returned — i.e. after the human already knew.

Continuwuity runs locally on the same box, so a Matrix alert still delivers on battery, to any device on the LAN or tailnet.

This makes the UPS migration a correctness fix, not tidiness. That alarm has never once worked during the event it exists for.

✅ The migration — what changed

All four Telegram alerters were moved onto the relay on 2026-09-07. Alert semantics were deliberately NOT changed — only the transport function.

The Telegram code was kept intact but inert, mirroring the bambuddy id=1 disabled-not-deleted precedent (2026-09-07-session-handover §3). Each script keeps its Telegram send function and its credentials; only the one line that selects the transport changed, so a revert is one edit rather than a rewrite. .bak backups sit alongside all five edited files.

ScriptRelay labelRollback — the exact line
/usr/local/bin/frigate-fps-watchdog.py[fps-watchdog]line 169: alert = send_matrix → alert = lambda text: send_telegram(token, chatid, text)
/etc/nut/nut-outage-handler.sh[ups]line 51 in send_alert(): send_matrix "$1" → send_telegram "$1"
/home/levander/top_kep_remind.py[top-kep]line 52 in send(): send_matrix(text) → send_telegram(text)
/home/levander/frigate-viewer-alert/{notify,watch}.py[viewer-alert]watch.py line 67: emit = send_matrix → emit = lambda text: send(token, chatid, text)

All four MATRIX_URL constants read http://127.0.0.2:8118/<label> — independently re-verified by grep on the box afterwards: no stray 8119 remains in any migrated script, 8119 is unbound, and the live relay listens on 127.0.0.2:8118 + 172.18.0.1:8118 under PID 1564638. 34 unit tests pass.

frigate-viewer-alert's trigger surface was NOT narrowed

It still fires on any tailnet request to Frigate, not only /live/ — proven during the migration by a delivery triggered by GET / rather than /live/. The §4 correction above still stands.

The live room was never touched

How the live room was protected — reuse this pattern

All testing ran against a throwaway relay on port 8119, with its own device and its own crypto store, since fully removed (device UrCaL98Ni3 logged out; @alerts now lists only the live uremzoqE3U | matrix-relay). The live relay — PID 1564638 — was never restarted. Proof: journalctl -u matrix-relay | grep -c 'sent event' → 0 across the entire migration window.

🟡 Still unproven — do NOT record these as done

  • 🟡 The :8118 production path was never exercised end to end. Everything was proven against :8119; the only delta was the port digit — now visually confirmed but not traffic-tested, because testing it would have posted into the live room. The first real alert is the test. First natural fire: the 20:00 top-kep daily reminder; viewer-alert fires whenever anyone next opens Frigate. UPS and fps-watchdog wait on genuine events.
  • 🟡 No real UPS event was simulated (ONBATT / ONLINE / LOWBATT). The send function was proven with the three real message strings, and the script was executed end to end through the safe unmapped COMMBAD branch.
  • 🟡 No unattended timer fire yet — the weekly timer was triggered manually via systemd.

🔴 top_kep_remind.py --daily was deliberately NOT run for real

Its daily path advances a watermark, so a “harmless” test run would have silently suppressed that evening’s 20:00 family reminder. Verified with --dry only; the watermark is still 1788717196. Rule: that script’s --daily path has a side effect — never test it live.

Also touched, and confirmed untouched

Touched beyond the four scripts: .bak backups (5 files) · two __pycache__ artifacts from syntax checks, both removed (the pre-existing internal-cam-alert.pyc left alone) · one benign line in /var/log/nut-outage.log from the safe execution test · test messages left in #relay-scratch and deliberately not redacted — that is the room’s stated purpose.

Confirmed untouched: the relay source · /etc/matrix-relay.env · config.yml (the telegram: block is intact) · cameras.exclude · telep-router · intruder-alarm.py (retired separately later the same day — see §6). No credential was rotated and no commit was made.

🔴 Gotcha — Continuwuity’s event lookup IGNORES the room in the URL

Hit for real during this migration, and it invalidates the obvious way to audit a room for test pollution.

GET /_matrix/client/v3/rooms/{room_id}/event/{event_id} on Continuwuity resolves purely by event ID — the room segment of the path is ignored. So /rooms/{scratch}/event/{id} cheerfully returns an event that lives in the live room, and vice versa.

A naive "is this event in room X?" check returns YES for any event that exists at all

It fails in both directions: it will report pollution that never happened, and it will equally hide a genuine live-room leak. This produced a false positive on the first room check during this session. Confirmed with a control: a known live-room event id, requested through the scratch-room path, returned successfully — with room_id set to the live room.

Reliable checks instead:

  • Read the returned event’s own room_id field and compare it to the expected room. That field is authoritative; the URL you requested is not.
  • To audit a room for pollution, do not use per-event lookups at all — page /rooms/{live}/messages?dir=b and inspect the timeline.
  • Or count at the source: journalctl -u matrix-relay --since '…' | grep -c 'sent event'.

This retroactively undermines any per-event audit of the "Telep Cam9" incident

The test-pollution remediation and the 41 redactions recorded in 2026-09-07-session-handover §5 / §6.3 turned on exactly this class of question. Anyone re-auditing that with per-event lookups would have got a wrong answer.

🔴 Gotcha — Tailscale SSH logs FULL command lines to the journal

tailscaled writes the entire command line of every Tailscale SSH invocation into the journal. It has both a debugging face and a security face.

Your own grep command becomes a match

Grepping the journal for a string finds the command you just ran, because that command text is in the journal too. Hit for real on 2026-09-07: a command containing the literal StartLimitIntervalSec, run to check whether that warning had stopped, matched itself — making an already-fixed problem look unfixed. Filter on the systemd[1]: prefix, or grep for the shape of the message rather than a bare keyword.

🔴 NEVER pass a secret as a command-line argument over Tailscale SSH

It lands in the journal in plaintext. This applies directly to the upcoming Telegram token rotation (see ⚠️ Rotate vs delete — they fail differently): set the new token by editing files or feeding it on stdin — never as an argument.

🔴 Silently broken — nobody is watching

Four paths that look like alerting and deliver nothing:

  • 🔴 smartd alerts go nowhere. /etc/smartd.conf uses -m root -M exec …/smartd-runner → /etc/smartmontools/run.d/10mail, but no MTA is installed at all — no sendmail, mail, mailx, msmtp, postfix or exim. /var/mail is empty and nothing appears in the journal. Given the earlier power event that damaged an NVMe (2026-08-05-power-root-cause-nvme-damage-ups-kb-handover), disk-health warnings landing in /dev/null is arguably the biggest hole on the box.
  • 🔴 netdata notifies nobody. All 28 SEND_* methods are set to "NO" — a deliberate override of the stock YES defaults. Consistent with the to: silent alarms recorded in 2026-08-31-telep-monitoring-and-file-services, but broader: no netdata alarm can reach anyone by any method.
  • 🔴 No OnFailure= on any of the ~70 custom systemd units. Nothing alerts when a service dies. Every “is it still running?” fact in this vault is a manual observation.
  • 🔴 casino-alert double-running — see §5 above.

🔴 The leaked Telegram token is in THREE locations

2026-09-07-session-handover open item 4 records only frigate-notify/config.yml. The real footprint:

LocationModeRead by
/home/levander/nvr/frigate-notify/config.yml—frigate-notify (disabled), top_kep_remind.py, frigate-viewer-alert, the duplicate top-kep-remind.py — intruder-alarm.py (retired 2026-09-07, §6)
/etc/nut/telegram.env0600 rootfrigate-fps-watchdog, nut-outage-handler.sh
/etc/casino-alert.sh (on telep-router)0755 — world-readablecasino-alert

The router copy is worse than what is recorded

The token is hardcoded in plaintext in a world-readable script on the router. Rotation must cover all three locations — a rotation that only touches the mainframe leaves a live token readable by any local user on the gateway.

⚠️ Rotate vs delete — they fail differently

A live footgun; the distinction matters and is not intuitive.

✅ Post-migration: rotating the token is now SAFE — deleting the block still is NOT

Updated 2026-09-07, after the migration above. All four host alerters now send via Matrix, so the Telegram path is inert rollback only. Rotate freely — including the world-readable router copy in /etc/casino-alert.sh, which is untouched and still needs rotating. 🔴 Deleting the telegram: block is still NOT safe. load_telegram_creds() is deliberately still called so the rollback stays available, so deletion would still crash-loop frigate-viewer-alert. Rotate; do not delete. 🔴 Two things now make a careless rotation dangerous, and they compound. (1) top_kep_remind is no longer the loud canary — it is on Matrix, so a missed location fails completely silently, with no systemctl --failed entry to catch it. (2) 🔴 Tailscale SSH logs full command lines to the journal, so passing the new token as an argument leaks it in plaintext — see 🔴 Gotcha — Tailscale SSH logs FULL command lines to the journal. Edit files or use stdin, and verify all three locations by hand. Everything below is retained as the pre-migration behaviour record — and it still describes exactly what happens to any script that gets rolled back onto Telegram.

Rotating the token → every alerter keeps running and silently posts to a dead token:

AlerterBehaviour after rotation
frigate-viewer-alert🔇 silent
frigate-fps-watchdog🔇 silent
nut-outage-handler.sh🔇 silent
casino-alert🔇 silent
top_kep_remind.py🔊 LOUD — urlopen raises HTTPError 401, no try/except, the unit exits non-zero and shows up in systemctl --failed

Deleting the telegram: block from frigate-notify/config.yml → load_telegram_creds() raises ValueError at startup → frigate-viewer-alert crash-loops under Restart=always.

Why the silence — two different mechanisms

  • frigate-fps-watchdog and nut-outage-handler.sh shell out to curl -s without -f, so curl exits 0 on HTTP 401 and the script logs "telegram sent" regardless. nut-outage-handler.sh discards the response body, making the failure undetectable from its own log.
  • frigate-viewer-alert is silent for a different reason: a caught exception that prints only to the journal.

Consequence — the token block must STILL not be deleted, even though the scripts are migrated

Updated 2026-09-07. The scripts that read it have been migrated, but they still call load_telegram_creds() on purpose to keep the rollback path alive — so deleting the block still crash-loops frigate-viewer-alert. Current readers: top_kep_remind.py and frigate-viewer-alert (both deliberately kept for rollback) plus the duplicate top-kep-remind.py. intruder-alarm.py was one of them and is now retired (§6) — that changes the reader count, not the verdict. Rotation is now unblocked and safe, but a missed location will not announce itself — and top_kep_remind, the one canary you had, no longer raises on a dead token either, because it is on Matrix now. Rotate all three locations in one pass.

Ruled out — checked, and NOT alert producers

Recorded so nobody re-checks them:

  • print-guard (/opt/print-guard/print-guard, 290 lines) — talks only to the bambuddy API at http://127.0.0.2:8000: GET on queue/printers, PATCH …/{id} {"manual_start": true}. No Telegram, Matrix, webhook or mail call anywhere. Its notifications ride bambuddy, which is already on Matrix.
  • tv-presence (/opt/tv-control/presence, 241 lines) — polls the router’s APs over SSH (root@192.168.1.1, key ~/.ssh/router_alarm, phy0-ap0/phy1-ap0) plus arp-scan on enp5s0 for two tracked MACs, and drives /opt/tv-control/tv. No HTTP client at all. Its “arrival path” powers the TV on — it never messages anyone.
  • internal-cam-alert.py — draws an X11 banner on DISPLAY=:0. Zero network output.
  • telep-selftest — hits api.telegram.org only as a reachability probe (2026-08-15-admin-portal-passkey-olivetin). ⚠️ It will keep showing up in future Telegram greps — it is not an alerter.
  • thermalwatch, dnsmon, camwall-watchdog, camwall-frigate-watch, wifi-usage, power-restore.sh — all act locally, no channel.
  • net-monitor probe.sh / starlink_status.py / wifi_survey.py — CSV output only (2026-07-28-net-monitor, 2026-09-04-starlink-wan-migration-dish-telemetry).
  • continuwuity-backup.py — errors go to a .err file.
  • router cams-guard.sh — logger to syslog only.
  • Frigate itself — mqtt: enabled: false and no webhooks. It has no independent notification path; everything rides frigate-notify.

print-guard and tv-presence are NOT broken alerting

Both are carried in older open-item lists as “has never fired on real data” (2026-09-05-session-handover item 8, 2026-09-07-session-handover item 11). That is true of their action paths and is not an alerting defect — neither one has ever been able to send a message. They are not migration targets.

Reusable gotchas

  • curl -s without -f exits 0 on HTTP 4xx. Any script that treats curl’s exit code as delivery confirmation will log success while sending nothing. Add -f, or inspect the response body.
  • A “disabled but not deleted” rollback block can become load-bearing. frigate-notify’s telegram.enabled: false left token/chatid in place, and three unrelated scripts now read their credentials from it. Disabling a feature does not mean its config is unreferenced — grep before deleting.
  • Migrating to the relay needs no relay change. It accepts any path as a source label with title/message or raw text, so each alerter is a one-function URL swap to http://127.0.0.2:8118/<name>.
  • An alerter that shares a failure domain with the thing it monitors is not an alarm. UPS alerts over the WAN cannot survive a power cut. Keep the alert path local to the failure.
  • “Never seen it fire” is only as strong as the state you kept. casino-alert’s cooldown state lives in /tmp and resets on boot, so its observation window is worthless — unproven, not dead.
  • 🔴 Continuwuity’s event-by-id lookup ignores the room in the path. Verify with the event’s own room_id field, never the URL you requested — a per-event “is it in room X?” check is wrong in both directions. See the section above.
  • Migrating an alerter is a one-function URL swap — keep the old transport callable and record the exact rollback line, so reverting is one edit rather than a rewrite.
  • Test the new transport against a scratch relay instance, never the live one — separate device, separate crypto store, separate port, torn down afterwards. Do not restart the live relay.
  • 🔴 Some “test” invocations have side effects that suppress future real alerts — a watermark, a debounce, a rate-limit window. Check for state advancement before running an alerter for real. top_kep_remind.py --daily advances a watermark and would have silently eaten that evening’s reminder.
  • 🔴 Tailscale SSH logs the FULL command line to the journal. Two consequences: your own grep command shows up as a match (a fixed problem can look unfixed — filter on systemd[1]: or on the message shape), and any secret passed as an argument is written to the journal in plaintext — use file edits or stdin instead.
  • Retire by archiving, not deleting. intruder-alarm went to /root/retired/2026-09-07-intruder-alarm/ with its unit, its script and a README; restore is mv + daemon-reload. Grep for dependents first — and record the grep evidence, so the next agent does not re-derive it.

Graph View

Backlinks