Session Handover — 2026-09-07

For the next agent

Homelab session 2026-09-05 → 2026-09-07. The headline is a self-hosted Matrix homeserver (Continuwuity) on the tailnet, with every homelab alert rerouted off Telegram and into it, plus a restore-tested backup of that homeserver. Alongside that: a wedged smart plug, an AMS humidity-alert noise fix, two camera/alert findings, and research (no build) into a markdown renderer for the telep homepage. ✅ Both alert defects are now FIXED and verified — Frigate image attachments and bambuddy finish photos both land in the room. See §6, and note the corrected root cause: it was a concurrency race that killed the Matrix session mid-send, not the suspected malformed info block. 🟡 Still unproven: the live bambuddy print_complete trigger has never fired with a photo. The first real print completion confirms it. ⚠️ All 41 relay-sent events were redacted from the live alert room, so there is no alert history before this point — that is expected, not a failure. ✅ A full alert-source inventory was done on 2026-09-07 (2026-09-07-alert-source-inventory), and the four remaining Telegram alerters have since been MIGRATED onto the relay — frigate-fps-watchdog, the NUT outage handler, the top-kép reminders and frigate-viewer-alert, all verified and running (open item 12). 🟡 But the :8118 production path was never traffic-tested — everything was proven against a throwaway :8119 relay, so the first real alert is the test. The inventory also found smartd, netdata and every custom systemd unit alerting NOBODY, and that the leaked Telegram token sits in three locations, one of them world-readable on the router — ✅ rotating it is now safe; deleting the telegram: block still is not. There are still no off-site backups, and two leaked credentials remain unrotated. Those are the biggest standing risks.

Status: Between tasks, 2026-09-07. Predecessor: 2026-09-05-session-handover.

---

What we accomplished

1. Matrix homeserver — new, working

Continuwuity v26.8.1, image forgejo.ellis.link/continuwuation/continuwuity:v26.8.1, digest sha256:fdf3cd0f…, pinned.

• Compose project at /home/levander/matrix/, config continuwuity.toml, RocksDB data at /home/levander/matrix/data.
• Backend binds 127.0.0.2:8008 — 127.0.0.1 is refused. House convention, see tailnet-service-exposure-convention.
• Its own sidecar tailnet node chat (tag:telep, tailscaled-chat.service), served at https://chat.taild4189d.ts.net, tailnet only, never Funnel.
• Federation OFF, open registration OFF — both proven with live 403s, not just assumed from config.
• Accounts: @andras (admin), @phone, @mfalusi, @puliki, @alerts (bot), plus the built-in @conduit. Passwords in /root/matrix-credentials.txt (0600).

server_name = chat.taild4189d.ts.net is PERMANENT

It is baked into the database. Changing it means wiping everything. It was chosen deliberately over a custom domain because the server genuinely is at that hostname — so no .well-known delegation is needed and Tailscale supplies the TLS cert. Do not “improve” this later.

Large uploads are the whole point of the deployment. max_request_size = 536870912 (512 MiB, 25× the default) so uncompressed phone photos go through intact. This was proven end to end — a real 400 MiB upload returned HTTP 200 — and tailscale serve’s reverse proxy was verified capless by source inspection of ipn/ipnlocal/serve.go at v1.102.2: no MaxBytesReader, no LimitReader, no ContentLength gate, just a stock httputil.ReverseProxy.

Gotchas worth their weight

Do NOT install conduwuit — it is ARCHIVED

Continuwuity is the maintained community continuation; Tuwunel is a competing live fork. And the rename is only skin deep — expect all of this while reading logs and paths:

• binary /sbin/conduwuit · data dir /var/lib/conduwuit · log modules conduwuit_* · admin bot @conduit:… · example config conduwuit-example.toml

• The image is distroless — no shell, no curl. So no docker exec debugging and no meaningful healthcheck. Everything is diagnosed from logs and from the outside.
• allow_announcements_check and allow_check_for_updates are aliases of the same field. Setting both is a fatal duplicate field error that crashloops the container.
• /.well-known/matrix/client 404s unless [global.well_known] client is set explicitly. It is NOT derived from server_name.
• QR sign-in (MSC4108) is NOT implemented in v26.8.1. /_matrix/client/v1/rendezvous and both unstable paths return 404, and there is no rendezvous code in the source tree. The token half (MSC3882, get_login_token: true) and full OAuth 2.0 are implemented. Re-check after a version bump.
• ✅ OAuth browser sign-in DOES work and is the practical answer. auth_metadata advertises authorization_code, refresh_token and device_code grants at /_continuwuity/oauth2/*. Both Element Desktop and Element X on iOS handed off to a browser instead of showing an in-app password form — confirmed in the server log (Issuing OAuth authorization code client_name="Element").

The Admin Room must stay UNENCRYPTED — permanently

The @conduit bot has to read !admin commands in the clear. Room encryption in Matrix is one-way, so enabling it in the Admin Room would permanently break user management with no way back.

The mainframe CANNOT reach the homeserver — self-verification is impossible

The tailnet ACL blocks tag:telep → these sidecar nodes on TCP/443, so telep-mainframe and telep-router cannot fetch https://chat.taild4189d.ts.net at all. chatcut and drive fail identically — this is the ACL, not a broken service. Test from a phone or laptop, or against the loopback backend 127.0.0.2:8008.

User management happens in the Admins room: !admin users create-user <name>, reset-password, list-users, make-user-admin, deactivate, suspend, logout.

create-user / reset-password print the password in PLAINTEXT into the room

And it persists in the database. Redact the message after copying the password.

2. Matrix backups — new, working, restore-tested

Script /usr/local/sbin/continuwuity-backup.py, run from root’s crontab at 30 3 * * * — deliberately offset from bambuddy’s 03:00 job. Destination /home/levander/backups/continuwuity/, directory 0700, archives 0600, 7-day retention with pruning.

Strategy: native online RocksDB backup, zero recurring downtime. Triggered non-interactively by SIGUSR2, wired up with admin_signal_execute = ["server backup-database"] in the config. Verified empirically, not assumed: docker kill -s SIGUSR2 → Created database backup #1 … in 47 files, no admin room, no human in the loop.

Two findings that would have made a naive backup UNRESTORABLE

1. backup-database produces a RocksDB BackupEngine store (meta/, private/, shared_checksum/), not an openable database — and there is no restore admin command. The script therefore materialises it back into a plain db/, so recovery is just tar xzf + cp with no tooling required.
2. The native backup covers only the database. Without media/ and archive/ the server refuses to start: “Critical error starting server: Failed to verify media integrity.” Both are now included.

The restore was actually rehearsed, not assumed: extracted → booted a throwaway container → /_matrix/client/versions returned 200 → users list-users returned all five accounts.

The "76 MB" data dir is a du artifact

Real size is 1.4 MB. du counts RocksDB’s preallocated WAL and MANIFEST. Archives land at ~222 KB.

3. Alerts rerouted: Telegram → Matrix — working, images included

Alert room telep-ertesitesek — #telep-ertesitesek:chat.taild4189d.ts.net / !gCuOI7uLN2JNqp5XR2voIsisfO0CXxBECMBh7CqynqQ. Encrypted (m.megolm.v1.aes-sha2). Members: @alerts, @andras, @phone joined; @mfalusi, @puliki invited only.

Relay: /opt/matrix-relay/matrix-relay + matrix-relay.service. matrix-nio 0.26.0 / vodozemac 0.10.0 — nio 0.26 dropped libolm for the Rust vodozemac backend, so the installed libolm3 is present but unused. Key store /opt/matrix-relay/store (0700); config /etc/matrix-relay.env (0600).

It binds 127.0.0.2:8118 AND 172.18.0.1:8118. The second bind is required because frigate-notify sits on the nvr_default docker bridge and cannot reach host loopback. Verified not tailnet-exposed: the node’s tailnet IP refuses 8118, no tailscale serve mount references it, and 172.18.0.0/16 is not among the advertised routes.

Two real bugs found and fixed during the build

1. Undecryptable messages. nio only re-shares a megolm session when it expires, so a member who joined later never received keys. Fixed by refreshing device lists and rotating the session when the member-device set changes. ignore_unverified_devices=True is the correct API for 0.26.
2. Crash loop on restart. With a persisted sync token the incremental sync returns no rooms, so the join check failed forever. Fixed by forcing a full sync at startup.

Final routing state:

Source	Telegram	Matrix webhook
bambuddy	provider id=1 DISABLED but NOT deleted (rollback path)	provider id=2 enabled, 15 events, mirroring id=1
frigate-notify	disabled	enabled
intruder-alarm.py	✅ RETIRED 2026-09-07 — archived to /root/retired/2026-09-07-intruder-alarm/, not deleted	— (never portable)

frigate-notify has a native matrix backend — it was deliberately NOT used

That container has no persistent volume beyond config.yml, so its crypto store would be wiped on every restart, producing recurring “unable to decrypt”. One E2EE identity with one persistent store is the safer design. See also 2026-07-28-frigate-notify-camera-exclude.

Link-rewriting bug worth recording

Payload URLs use the container-internal host http://frigate:5000, which the relay cannot resolve. They must be rebuilt against the public tailnet URL or the room fills with useless links.

The full alert-source inventory now lives in 2026-09-07-alert-source-inventory

A source-level audit on 2026-09-07 found six alert producers in total. The two in the table above were already migrated. ✅ The other four — frigate-fps-watchdog, the NUT outage handler, the top-kép reminders and frigate-viewer-alert — were migrated later the same day (open item 12), each a one-line transport swap with the Telegram code kept intact but inert. 🟡 The :8118 production path is still untested by real traffic — see the inventory note’s migration section. Two are not migratable as things stand: router casino-alert is infra-blocked (the relay binds 127.0.0.2 + 172.18.0.1, neither reachable from telep-router) and is running twice, so every hit would double-send; intruder-alarm is an interactive bot — send_photo, edit_message_text, inline keyboards, callback polling for MAC enrollment — and Matrix has no inline-keyboard equivalent, so porting it is a rewrite, not a migration. ✅ intruder-alarm has since been RETIRED (2026-09-07) — archived, not deleted: /root/retired/2026-09-07-intruder-alarm/ holds the unit, the script and a README, and restore is mv both back + daemon-reload. systemctl status now returns “Unit intruder-alarm.service could not be found”. Nothing depended on it — grep-verified; the only hit outside its own files was a comment in /home/levander/ruview/scripts/c6-presence-watcher.py plus two coincidental word matches in ruview/examples/research-sota/. 🧹 Side effect: its unit’s Unknown key 'StartLimitIntervalSec' in section [Service] warning on every daemon-reload is gone — and there is no wider misconfiguration to chase, since camwall.service, camwall-x.service and nvidia-cdi-refresh.service all have that key correctly in [Unit]. ⚠️ print-guard and tv-presence are NOT alert producers and never were — see open item 11.

🔴 The headline finding — Telegram fails exactly when it is needed

31 of 34 UPS alert sends FAILED, and every single failure falls inside the 2026-08-07 (6 failures) and 2026-08-18 (25 failures) outage windows. The cause is structural, not flaky: the power was out, so the WAN was down, so the alert could not leave the building. The only sends that succeeded landed after power and network returned — i.e. after the human already knew. Continuwuity runs locally on the same box, so a Matrix alert still delivers on battery, over the LAN and tailnet. 🔴 This makes the UPS migration a correctness fix, not tidiness — that alarm has never once worked during the event it exists for. Generalise it: an alerter that shares a failure domain with the thing it monitors is not an alarm. Keep the alert path local to the failure.

4. Smart plug + AMS alert noise

• Tapo P115 at 192.168.1.167 wedged. Associated at WiFi with excellent radio (−39 dBm, SNR 64) and a valid DHCP lease, but dead at IP level — ping 100% loss, ARP STALE, port 80 closed. Classic wedged IoT network stack; the fix is a physical power-cycle. The bridge behaved correctly throughout: fast 503s with backoff, NRestarts=0, no hang. Nothing depends on it (auto_on/auto_off both false).
• AMS humidity alerts were 32 of the last 40 notifications — hourly, forever. Cause: PC’s threshold was 30 while the AMS runs at 37–42%, and the AMS physically cannot dry to 30% (65 °C ceiling; PC needs 80 °C). The alert was correct and unactionable — the worst kind. Background in 2026-09-03-bambuddy-preheat-chamber-target-bug-and-clog-rca.

Key insight — per-material humidity thresholds do NOT work on a single-chamber AMS

There is one sensor for all four slots, so whichever material has the lowest threshold alerts permanently, regardless of what is actually loaded. The entire per-material map was replaced with {"default": 50}, which means “the AMS is wetter than it normally runs” — and that is actionable (exhausted desiccant, wet spool, humid weather). The PC-drying knowledge now lives where it belongs: the PC / PC-FR Dryness Check maintenance task (14 days) and the PC-FR pipeline description.

5. Camera / alert findings

• frigate-notify excludes telep_cam3 and telep_cam4 via cameras.exclude. This is a deliberate, user-confirmed setting — those cameras produce no notifications at all. Do not “fix” it. (2026-07-28-frigate-notify-camera-exclude)
• A genuine person was detected on cam3 at 02:27 and 02:37 (79% confidence, verified by viewing the snapshot — a real person by the railing at night). No alert fired, because of the exclusion above. That trade-off is now a known, accepted consequence.
• Real detections do get images: 4 of 5 frigate posts in the sample had image=True (85,900 and 77,374 byte JPEGs). The only image-less one was the synthetic test below.

A synthetic test alert was posted into the LIVE alert room

“Telep Cam9”, event id 9999999999.000000-nope, injected by a subagent testing the snapshot-failure path. It confused the user into thinking a real detection had failed. Lesson: test alerts belong in a scratch room, or must be redacted afterwards. ✅ Both have since been done — a dedicated scratch room #relay-scratch:chat.taild4189d.ts.net now exists for exactly this (§6.3), and every relay-sent event was redacted from the live room. ⚠️ That redaction swept up the real ones too — see §6.3.

6. The two alert defects — RESOLVED and verified

This section previously read "IN PROGRESS — do NOT record these as done"

Both defects are fixed, deployed and verified against real traffic. The only thing still unproven is the live bambuddy print_complete trigger — see §6.2’s caveats and open item 1.

6.1 Frigate snapshot images not rendering — FIXED

The original hypothesis was WRONG — this correction matters more than the fix

The suspected incomplete info block (mimetype / w / h / size) was NOT the cause. The m.image event was spec-compliant all along: correct msgtype, non-empty body, complete info, and a well-formed encrypted file block (v: "v2", key alg A256CTR, iv, hashes.sha256, mxc:// url) with no stray plaintext url. No thumbnail_file is needed — Element renders encrypted images without one. Anyone re-reading the old theory should stop chasing the event shape.

Real root cause: a concurrency race between the HTTP thread and sync_forever.

The relay’s _prepare() called add_changed_users() + keys_query() from the HTTP thread while matrix-nio’s sync_forever was consuming the same shared users_for_key_query set. Whichever consumed it first left the other calling keys_query() with an empty set, which raises LocalProtocolError: No key query required. — and that kills the Matrix session and closes the aiohttp client between the text send and the image send. The text arrived; the image was lost. Being timing-dependent and intermittent, aggregate stats looked healthy.

03:58:43 sent event $kV3Y...                                    (text OK)
03:58:43 matrix session ended: LocalProtocolError: No key query required.
03:58:43 image send failed: RuntimeError: Session is closed     (image LOST)

Fixes applied:

• Stopped mutating nio’s shared key-query state. Device freshness now comes from read-only inspection of device_store; the signature-based megolm rotation from §3 is unchanged. room_send already handles members_synced / keys_query internally.
• Text and image now share one coroutine, up to 3 attempts, with the text tracked so a retry never duplicates it — only the missing part is re-sent.
• Fallback chain snapshot.jpg → thumbnail.jpg → /api/<camera>/latest.jpg, and it no longer skips when has_snapshot is false. Proven: a bogus event id 404’d on both of the first two and still recovered a 106 KB frame from latest.jpg.

Verification: zero session ended / Session is closed events since the fix, plus live proof — a genuine person detection on telep_cam3 at 03:11:22, decrypted as @phone: 162,321 bytes, valid JPEG 1280×720, complete info and file.

Why a telep_cam3 alert exists at all when §5 says cam3 never notifies — both facts are true

cameras.exclude is a frigate-notify setting, and frigate-notify sits UPSTREAM of the relay. The relay has no camera filtering of its own — it posts whatever arrives on its webhook. So an event POSTed directly at the relay on 127.0.0.2:8118 bypasses the exclusion entirely, because the exclusion never gets a chance to run. That is exactly what happened here: the fixing agent replayed a genuine cam3 detection straight at the relay to exercise the snapshot fetch/upload path. cameras.exclude really was untouched (its regression check is correct) and cam3 really does still produce no notifications through the normal frigate-notify path (§5 is correct) — the 03:11:22 verification simply never travelled that path. Corroborated by the fixing agent’s own account: it redacted the real-sender verification alerts “since a cam3 alert would itself have been confusing given cam3 is excluded”. ✅ CONFIRMED on 2026-09-07 by reading the relay source — this is no longer an inference. The relay computes source = parts.path.strip("/").split("/")[0], so the request path is nothing but a LABEL; render_payload falls back to the title / message keys and then to raw text, and decorate prefixes [source]. It accepts a POST on ANY path and posts whatever it receives. There is no camera filter — and no filter of any kind. Full audit: 2026-09-07-alert-source-inventory.

The generalisable footgun — filtering in frigate-notify does NOT protect the relay

cameras.exclude is enforced upstream; the relay posts anything it receives on :8118. Any direct POST — testing, replay, or a future integration — reaches the LIVE alert room regardless of camera. If a camera must never appear in a room, the relay needs its own filter; the exclude alone is not a guarantee. This is precisely how test traffic reaches the live room — the same failure mode as the “Telep Cam9” incident in §5, and the reason #relay-scratch:chat.taild4189d.ts.net exists (§6.3).

🔴 The consequence is broader than cameras — the relay is UNAUTHENTICATED and UNVALIDATED

Source-confirmed: no auth, no path validation, no content validation on either bind. Listening state verified with ss -lntp — exactly 127.0.0.2:8118 and 172.18.0.1:8118, one pid; the tailnet IP (100.115.209.87) is not bound, so this is not tailnet-exposed. 🔴 But 172.18.0.1:8118 is the nvr_default docker bridge gateway, which means ANY container on that bridge can post arbitrary content into the LIVE alert room. That bind is required for frigate-notify to reach the relay (§3), so it cannot simply be removed — but the exposure is real and should be closed with auth or a shared-secret header on the relay, not by trusting the bridge. ↔️ The flip side is a convenience: migrating an alerter to the relay needs no relay change at all — any path works as a source label, so each alerter is a one-function URL swap to http://127.0.0.2:8118/<name>.

6.2 bambuddy finish photos not attached — IMPLEMENTED

• The payload key is finish_photo_url, and it is a relative path: /api/v1/archives/{archive_id}/photos/finish_<ts>_<hash>.jpg. It is present on print_complete, print_failed and print_stopped.
• The two-image distinction is now confirmed, not assumed: /print-log/{id}/thumbnail and /archives/{id}/thumbnail both return the slicer render (512×512 PNG). The finish photo — the actual camera capture — exists only under /archives/{id}/photos/{filename}. That is the one attached.
• Same single code path as frigate: resolve_media → Fetcher → encrypted upload → m.image. PNG magic-byte / dimension support was added so the path is no longer JPEG-only.
• Proof, using a real photo from archive 94, decrypted as @phone: body: print-finish-3dp-093-310-suzuki-keychain.jpg, info: {mimetype: image/jpeg, size: 424330, w: 1680, h: 1080}, SHA256 byte-identical to the source — b861624248f8550ce030ad279429e39c42e8bc5a8cf3fdaaa91f4f08c2a8588a.
• finish_photo_url is now suppressed from the alert text, since the photo itself is attached.

Still unproven — do NOT flatten these into "done"

• 🟡 The live print_complete trigger has never fired with a photo. bambuddy’s provider test endpoint sends a generic payload with no finish_photo_url, so the real event could not be fired without an actual print. The fetch → encrypt → upload → decrypt path was proven byte-identical against a real finish photo from a past print, but the live hook firing remains untested. The first real print completion confirms it.
• 🟡 print_failed / print_stopped finish photos are inferred from the same key — neither was fired.
• 🟡 Only the car and person label emoji were exercised.
• 🟡 The Frigate latest.jpg fallback derives the camera slug from the display name (Telep Cam1 → telep_cam1). That holds for the current cameras, but a camera renamed to something non-matching would silently skip that last fallback — the alert text still sends.

6.3 Test-pollution remediation — and the scratch room

All synthetic traffic ran through an isolated scratch instance: its own @alerts device, its own crypto store, port 8119, room #relay-scratch:chat.taild4189d.ts.net. That instance is now stopped, its device logged out, its files removed — port 8119 is gone.

#relay-scratch:chat.taild4189d.ts.net is the designated home for test alerts

The scratch ROOM was deliberately retained. This is the concrete answer to the “Telep Cam9” incident in §5 — send test alerts there, never into telep-ertesitesek.

⚠️ ALL 41 relay-sent events were REDACTED from the live telep-ertesitesek room

That includes two real-sender verification alerts. The room now holds zero un-redacted relay messages, and this is irreversible: there is no alert history before this point. Flagging it loudly because a future agent looking at an empty room will otherwise conclude that alerts were never delivered — they were.

🔴 Do NOT re-audit this with per-event lookups — they cannot tell rooms apart

Continuwuity’s GET /_matrix/client/v3/rooms/{room_id}/event/{event_id} ignores the room segment of the path and resolves purely by event ID, so a “is this event in the live room?” check returns YES for any event that exists at all. It is wrong in both directions — it invents pollution that never happened, and it hides genuine leaks. Hit for real on 2026-09-07 and confirmed with a control. Check the event’s own room_id field, page /rooms/{live}/messages?dir=b, or count with journalctl -u matrix-relay | grep -c 'sent event'. Full write-up: 2026-09-07-alert-source-inventory.

The pre-change relay is backed up at /root/.mrelay/matrix-relay.bak3.

Regression checks, all clean: binds 127.0.0.2:8118 + 172.18.0.1:8118 with 127.0.0.1 refused (exit 7) · Restart=always / RestartSec=10, enabled + active · env 0600, store 0700, token absent from the journal · bambuddy id=2 enabled (15 events), id=1 Telegram disabled but not deleted · frigate-notify telegram: false, webhook: true · cameras.exclude untouched (telep_cam3, telep_cam4) · never-drop verified.

7. Knowledgebase / homepage context — researched, nothing built

A markdown renderer for “the telep homepage” was requested. Research was done; nothing was implemented, and the content to render was never specified.

• The existing renderer is the knowledgebase (2026-07-24-knowledgebase): Astro 5.14 + Starlight 0.36, built with Bun, project /home/levander/kb-astro/, canonical builder build_content.py — NOT convert.py, which is partial and produces a stale site (independently re-confirmed by a second research pass). Deploy via kb_build.sh → atomic swap into ~/knowledgebase/site/ (1.6 GB built). Served by knowledgebase.service (Flask + waitress) on 127.0.0.2:8092, tailnet https://knowledgebase.taild4189d.ts.net.
  • ⚠️ The knowledgebase is NOT containerised — it is a plain systemd service. The only related container is kb-qdrant, which earlier versions of this doc never mentioned at all.
  • ⚠️ The sidecar unit / socket / statedir are named kb while the hostname is knowledgebase. History: mkdocs-material → Astro (Aug 2026), done specifically to preserve byte-identical directory URLs.
• ✅ CONFLICT RESOLVED: home.taild4189d.ts.net fronts gethomepage (→ 127.0.0.2:3010), not the old static home-portal. Older vault notes claiming otherwise are stale.
  • 🟡 But the old static portal is STILL RUNNING and orphaned — python3 on 127.0.0.1:8093, pid 2716, serving a 3.8 KB August index.html that nothing proxies to. Worth killing — and note it sits on 127.0.0.1, exactly the address the hardening sweep moved services off.
• gethomepage cannot host arbitrary pages — it is a fixed dashboard. The proven mechanism for arbitrary UI is the iframe service widget (as already used for the TV control buttons, see 2026-09-05-session-handover).

Three gethomepage gotchas

• docker restart homepage does NOT apply config changes. Use curl http://127.0.0.2:3010/api/revalidate.
• Quote every description:. An unquoted colon-space blanks the entire dashboard.
• services.yaml has concurrent writers. Surgical line insertion only — never a YAML round-trip.

---

Where we paused / open items

Not done yet

1. 🟡 Both alert defects are FIXED (§6) — but the live bambuddy print_complete trigger has never fired with a photo. bambuddy’s provider test endpoint sends a generic payload with no finish_photo_url, so it could not be fired without a real print; the fetch → encrypt → upload → decrypt path was proven byte-identical against a past print’s finish photo. The first real print completion confirms it. Also never fired: print_failed / print_stopped photos (inferred from the same key), and only the car / person label emoji were exercised.
2. 🔴 There are NO off-site backups. Matrix backups, bambuddy backups and the live data all sit on the same 3.6 T LVM volume. Decision made: restic → Cloudflare R2 with client-side encryption. Current state: rclone v1.60 is installed but only has a gdrive remote; restic 0.18.0 is available in Debian 13’s repo but is NOT installed. Needs from the user: an R2 bucket, an S3-compatible API token (access key + secret + account id), and a restic repository password stored in a password manager. ⚠️ That password is unrecoverable — lose it and the backups are permanently unreadable.
3. 🔴 /opt/matrix-relay/store is backup-critical. If it is lost, every previously-sent alert becomes permanently undecryptable for everyone. Add it to the backup set.
4. 🔴 Rotate the Telegram bot token — and it lives in THREE places, not one. It was leaked into a chat transcript twice. ✅ Updated 2026-09-07 — rotation is now SAFE and unblocked. All four host alerters have been migrated to Matrix, so the Telegram path is inert rollback only and rotating breaks nothing anybody is watching. (The earlier wording “Telegram is disabled so it carries no alerts” was only ever true of frigate-notify — until the migration, four other alerters were live on this same token. See 2026-09-07-alert-source-inventory.) Locations: /home/levander/nvr/frigate-notify/config.yml (plaintext) · /etc/nut/telegram.env (0600 root, shared by frigate-fps-watchdog and nut-outage-handler.sh) · 🔴 hardcoded in plaintext in /etc/casino-alert.sh on telep-router, mode 0755 — WORLD-READABLE. That third copy is worse than anything previously recorded here. Rotation must cover all three. Also rotate the bambuddy API key, leaked earlier (carried over from 2026-09-05-session-handover).
   • 🔴 Before rotating, read this: 🔴 Tailscale SSH logs the FULL command line to the journal, so passing the new token as an argument writes it there in plaintext — edit files or feed it on stdin, never as an argument. And because top_kep_remind is no longer the loud canary (below), a missed location fails silently — so a careless rotation can both leak the new token and quietly leave a live one behind. Verify all three locations by hand.
   • ⚠️ Rotate ≠ delete, and they fail differently. Rotating leaves every alerter running and silently posting to a dead token — silent for frigate-viewer-alert, frigate-fps-watchdog, nut-outage-handler and casino-alert; LOUD for top_kep_remind only (urlopen raises HTTPError 401, no try/except ⇒ the unit exits non-zero and appears in systemctl --failed). Deleting the telegram: block from frigate-notify/config.yml makes load_telegram_creds() raise ValueError at startup ⇒ frigate-viewer-alert crash-loops on Restart=always.
   • 🔴 Post-migration status (2026-09-07): rotation can proceed NOW and safely — but the telegram: block must STILL NOT be deleted. The migrated scripts deliberately keep calling load_telegram_creds() so their rollback path stays available, so deletion still crash-loops frigate-viewer-alert. ⚠️ And top_kep_remind is no longer the loud canary — it is on Matrix now, so a missed rotation location will be entirely silent. Rotate all three in one pass. 🧹 Reader count changed, verdict did not: intruder-alarm.py was one of the telegram: block’s readers and is now retired; the remaining readers are top_kep_remind and frigate-viewer-alert (both deliberately kept for rollback) plus the duplicate top-kep-remind.py.
5. 🔴 A live @phone access token sits at /root/.mrelay/phone-session.json (0600) — a subagent’s decryption test harness with no ongoing purpose. Log the device out and delete the file.
6. 🔴 bambuddy’s API is unauthenticated with an empty users table. Anyone reaching bambuddy.taild4189d.ts.net has full control of a machine that heats to 280 °C. The tailnet ACL is the only gate.
7. 🟡 Tapo P115 needs a physical power-cycle (§4) — wedged at IP level despite a healthy radio link.
8. 🟡 @mfalusi and @puliki have not joined the alert room and have no E2EE devices — they cannot decrypt anything sent before they log in. This is inherent to Megolm, not a bug to chase.
9. 🟡 The markdown renderer was never built, and the content to render was never specified (§7).
10. 🟡 Kill the orphaned home-portal on 127.0.0.1:8093 (pid 2716).
11. Carried over and still open from 2026-09-05-session-handover: TV Auto Power Off is STILL not disabled; the Starlink mount is printed but not installed (obstruction was ~3.3%, see 2026-09-04-starlink-wan-migration-dish-telemetry); the north camera’s multipath (8.6–11 Mbit/s at good RSSI) needs a closer AP or a directional antenna; and print-guard’s guard path and tv-presence’s arrival path have STILL never fired on real data — ⚠️ but read that last one correctly: it is about their ACTION paths and is NOT an alerting defect. Neither is an alert producer at all — source-verified 2026-09-07: print-guard (/opt/print-guard/print-guard, 290 lines) talks only to the bambuddy API on http://127.0.0.2:8000 (GET queue/printers, PATCH …/{id} {"manual_start": true}) with no Telegram, Matrix, webhook or mail call anywhere, and tv-presence (/opt/tv-control/presence, 241 lines) has no HTTP client at all — it SSHes to the router’s APs (phy0-ap0/phy1-ap0) plus arp-scan and then drives /opt/tv-control/tv, so its “arrival path” powers the TV on and messages nobody. Neither is a migration target; neither is broken alerting. See 2026-09-07-alert-source-inventory.
12. ✅ DONE — the four remaining Telegram alerters are migrated onto the relay (frigate-fps-watchdog → [fps-watchdog], the NUT outage handler → [ups], the top-kép reminders → [top-kep], frigate-viewer-alert → [viewer-alert]), all verified and running. Telegram code kept intact but inert, mirroring bambuddy’s id=1 precedent; .bak backups beside all five edited files; the exact one-line rollback for each is recorded in 2026-09-07-alert-source-inventory. Alert semantics unchanged, 34 unit tests pass, and the live relay (PID 1564638) was never restarted — journalctl -u matrix-relay | grep -c 'sent event' → 0 across the whole window, because all testing ran on a throwaway :8119 relay with its own device and crypto store, since removed.
    • 🟡 Still unproven — the :8118 production path was never exercised end to end. The only delta from what was tested is the port digit, grep-confirmed but not traffic-tested (testing it would have posted into the live room). The first real alert is the test: the 20:00 top-kep reminder, and viewer-alert whenever anyone next opens Frigate. UPS and fps-watchdog wait on genuine events.
    • 🟡 No real UPS event simulated (ONBATT/ONLINE/LOWBATT) — the send function was proven with the three real message strings and the script run end to end via the safe unmapped COMMBAD branch. No unattended timer fire yet either; the weekly timer was triggered manually.
    • 🔴 top_kep_remind.py --daily was deliberately NOT run for real — it advances a watermark and would have silently suppressed that evening’s 20:00 family reminder. --dry only; watermark still 1788717196.
13. 🔴 Alert paths that are wired to NOTHING — nobody is watching. smartd sends to a nonexistent MTA (/etc/smartd.conf → 10mail, but no sendmail/mail/mailx/msmtp/postfix/exim is installed, /var/mail empty) — disk-health warnings go to /dev/null, which given the earlier NVMe damage is arguably the biggest hole on the box; netdata notifies nobody (all 28 SEND_* methods set to "NO"); no OnFailure= on any of the ~70 custom systemd units, so nothing alerts when a service dies; and router casino-alert is running TWICE (two PIDs, each with its own tail -F ⇒ every hit double-sends). See 2026-09-07-alert-source-inventory.

---

Key endpoints, paths & gotchas

Endpoints (tailnet-only — see tailnet-service-exposure-convention):

Service	URL
Matrix (Continuwuity)	https://chat.taild4189d.ts.net
Frigate	https://telep-mainframe.taild4189d.ts.net
Homepage dashboard	https://telep-mainframe.taild4189d.ts.net:8450 · https://home.taild4189d.ts.net · LAN: http://home.telep.lan (Caddy on :80)
tv-http (TV control)	https://telep-mainframe.taild4189d.ts.net:8451
bambuddy (UI + /mcp)	https://bambuddy.taild4189d.ts.net
Knowledgebase	https://knowledgebase.taild4189d.ts.net
CAD / exports / o3dv / FreeCAD MCP	https://cad.taild4189d.ts.net (+ :8080 exports, :8090 o3dv, :8443 FreeCAD MCP)

127.0.0.2 port map (consolidated):

3001   bambu-studio-api (slicer sidecar)
3010   homepage                 -> tailnet :8450 / home.*
3080   freecad
8000   bambuddy (UNAUTHENTICATED)
8008   matrix (Continuwuity)    -> tailnet chat.*
8085   exports
8087   o3dv
8091   bambuddy-mcp-bridge
8092   knowledgebase            -> tailnet knowledgebase.*  (sidecar unit is named `kb`)
8102   tv-http                  -> tailnet :8451
8117   tapo-bridge
8118   matrix-relay             (+ 172.18.0.1:8118 for the nvr_default docker bridge)
9099   telep-kb-mcp
11000/11001/11002  nextcloud / tsauth-proxy / onlyoffice

Paths added this session:

• /home/levander/matrix/{continuwuity.toml,data,media,archive} · /root/matrix-credentials.txt (0600)
• /usr/local/sbin/continuwuity-backup.py · /home/levander/backups/continuwuity/ (0700)
• /opt/matrix-relay/{matrix-relay,store} · /etc/matrix-relay.env (0600) · /root/.mrelay/matrix-relay.bak3 (pre-image-attachment relay backup)
• /root/.mrelay/phone-session.json (0600) — ⚠️ to be deleted, see open item 5
• /home/levander/kb-astro/ (+ build_content.py, kb_build.sh) · ~/knowledgebase/site/

Standing house rules reinforced this session:

• 🔴 Bind 127.0.0.2, never 127.0.0.1. (2026-08-31-tailnet-plaintext-port-hardening, tailnet-service-exposure-convention)
• 🔴 NEVER run /etc/init.d/network reload|restart on telep-router — it drops the site and the tailnet SSH used to manage it.
• 🔴 busybox on the router LIES — timeout is missing, and nc -z / ping -M do give false negatives. Test from telep-mainframe.
• 🔴 Frigate stores recordings in UTC — local 07:23 lives in the 05/ hour directory.

New reusable gotchas from this session:

• conduwuit is archived; Continuwuity is the fork — but the binary, data dir, log modules and admin bot are all still named conduwuit/conduit.
• Distroless images cannot be docker exec’d and cannot have a meaningful healthcheck.
• Config aliases (allow_announcements_check / allow_check_for_updates) crashloop on duplicate field.
• A RocksDB BackupEngine store is not an openable database — materialise it, or your backup is a paperweight.
• matrix-nio only rotates megolm sessions on expiry — rotate manually when the member-device set changes, or late joiners get undecryptable messages.
• A persisted sync token makes incremental syncs return no rooms — force a full sync at startup.
• Never mutate matrix-nio’s shared users_for_key_query from another thread — sync_forever consumes it, and a double-consume makes keys_query() raise LocalProtocolError: No key query required., killing the session mid-send. Let room_send handle key queries; inspect device_store read-only.
• A dead Matrix session between two sends loses the second one SILENTLY — send related text + media in one coroutine, and track which part already landed so a retry does not duplicate it.
• bambuddy’s thumbnail endpoints return the SLICER RENDER, not the camera capture — the finish photo lives only under /archives/{id}/photos/{filename}.
• Filtering that lives in frigate-notify does NOT protect the relay. cameras.exclude is enforced upstream; the relay posts anything it receives on :8118, so any direct POST — testing, replay, a future integration — reaches the live alert room regardless of camera. If a camera must never appear in a room, the relay needs its own filter.
• One sensor per AMS chamber ⇒ per-material humidity thresholds are structurally broken.
• tailscale serve imposes no upload size cap (verified against serve.go at v1.102.2).
• curl -s without -f exits 0 on HTTP 4xx — any script that treats curl’s exit code as delivery confirmation will log success while sending nothing. Add -f, or inspect the response body. (This is exactly why a Telegram token rotation would be silent for frigate-fps-watchdog and nut-outage-handler.)
• A “disabled but not deleted” rollback block can become load-bearing. frigate-notify’s telegram.enabled: false left token/chatid in place — and three unrelated scripts now read their credentials from it. Disabling a feature does not mean its config is unreferenced: grep before deleting.
• An alerter that shares a failure domain with the thing it monitors is not an alarm — UPS alerts over the WAN cannot survive a power cut. Keep the alert path local to the failure.
• The relay accepts ANY path as a source label (title/message keys, else raw text), so migrating an alerter needs no relay change — it is a one-function URL swap to http://127.0.0.2:8118/<name>. The same property is why it is unauthenticated and unvalidated (§6.1).
• “Never seen it fire” is only as strong as the state you kept — casino-alert’s cooldown state lives in /tmp and resets on boot, so its observation window proves nothing. Unproven, not dead.
• 🔴 Continuwuity’s /rooms/{room_id}/event/{event_id} IGNORES the room in the path — it resolves purely by event ID, so /rooms/{scratch}/event/{id} happily returns an event that lives in the live room. A naive “is this event in room X?” check therefore returns YES for anything that exists, and is wrong in both directions: it invents pollution that never happened and hides a genuine live-room leak. Read the event’s own room_id field (authoritative), page /rooms/{live}/messages?dir=b to audit a room, or count at the source with journalctl -u matrix-relay | grep -c 'sent event'. ⚠️ Directly relevant to §5/§6.3 — a per-event audit of the “Telep Cam9” pollution or the 41 redactions would have given a wrong answer.
• Migrating an alerter is a one-function URL swap — keep the old transport callable and record the exact rollback line, so reverting is one edit rather than a rewrite. (Same disabled-not-deleted shape as bambuddy’s id=1.)
• Test a new transport against a SCRATCH relay instance, never the live one — separate device, separate crypto store, separate port, torn down afterwards, and never restart the live relay. Verify with grep -c 'sent event' on the live unit’s journal.
• 🔴 Some “test” invocations have side effects that suppress future REAL alerts — a watermark, a debounce, a rate-limit window. Check for state advancement before running an alerter for real. top_kep_remind.py --daily advances a watermark and would have silently eaten that evening’s 20:00 reminder; --dry is the safe path.
• 🔴 Tailscale SSH logs the FULL command line to the journal. tailscaled writes the entire invocation, which bites twice: your own grep command comes back as a match — a command containing StartLimitIntervalSec, run to check whether that warning had stopped, matched itself and made a fixed problem look unfixed (filter on the systemd[1]: prefix, or grep the message shape, not a bare keyword) — and 🔴 any secret passed as an argument is written to the journal in plaintext. ⚠️ Directly relevant to the pending credential rotation (open item 4): set the new token by editing files or via stdin, never as an argument.
• Retire by archiving, not deleting — intruder-alarm went to /root/retired/2026-09-07-intruder-alarm/ with its unit, its script and a README, so restore is mv + daemon-reload. Grep for dependents first and record the evidence, so nobody re-derives it later.
• StartLimitIntervalSec belongs in [Unit], not [Service] — a unit with it misplaced emits Unknown key … on every daemon-reload. Recurring journal noise of that shape is usually one bad unit, not a fleet-wide problem: camwall.service, camwall-x.service and nvidia-cdi-refresh.service all had it right.

Related notes: 2026-09-07-alert-source-inventory (alert audit from this session) · 2026-09-05-session-handover (predecessor) · 2026-09-04-starlink-wan-migration-dish-telemetry · 2026-09-03-bambuddy-preheat-chamber-target-bug-and-clog-rca · 2026-07-28-frigate-notify-camera-exclude · 2026-07-24-knowledgebase · 2026-08-31-tailnet-plaintext-port-hardening · tailnet-service-exposure-convention · telep-mainframe · telep-router

Not yet in dedicated notes

§1–§3 and §6 (the Matrix homeserver, its backup design, the Telegram → Matrix alert migration, and the media-attachment fix) have no standalone vault notes yet — this handover is currently the only record. All are substantial enough to deserve promotion, especially the backup restore procedure and the users_for_key_query race.

---

Continuation prompt (copy-paste to resume)

Pick up the 2026-09-07 homelab session (read projects/homelab/2026-09-07-session-handover.md, and use the historian for deeper context). Priorities: (1) Set up off-site encrypted backups: restic → Cloudflare R2 — restic 0.18.0 is in Debian 13’s repo but not installed; rclone v1.60 is installed but only has a gdrive remote. Ask the user for an R2 bucket, an S3-compatible API token (access key + secret + account id), and a restic repo password stored in a password manager — it is unrecoverable. Include /opt/matrix-relay/store in the backup set: lose it and every past alert is permanently undecryptable. (2) Rotate the leaked credentials — now unblocked. The Telegram bot token was leaked to a transcript twice and lives in THREE places: /home/levander/nvr/frigate-notify/config.yml, /etc/nut/telegram.env, and 🔴 hardcoded in world-readable /etc/casino-alert.sh on telep-router. ✅ All four host alerters are on Matrix now, so rotation is safe — but do NOT delete the telegram: block, still read for rollback, and note that a missed location will be completely silent (top_kep_remind is no longer the canary). Also rotate the bambuddy API key, and log out + delete the stray @phone token at /root/.mrelay/phone-session.json. (3) On the first real print completion, confirm the bambuddy finish photo actually attaches — the code path is proven byte-identical against a past print’s photo, but the live print_complete hook has never fired with a photo because bambuddy’s provider test endpoint sends a generic payload without finish_photo_url. (4) Confirm the four newly-migrated alerters actually deliver on :8118 — they were verified against a throwaway :8119 relay and the production port was never traffic-tested. First natural fires: the 20:00 top-kep reminder and the next Frigate page view (viewer-alert). Check the room, or journalctl -u matrix-relay | grep 'sent event'. Done and verified: the Continuwuity v26.8.1 homeserver at https://chat.taild4189d.ts.net (federation and open registration OFF, 512 MiB uploads proven), its restore-tested nightly backup (30 3 * * *), the Telegram → Matrix alert migration (bambuddy provider id=2, frigate-notify webhook, plus all four host alerters as of 2026-09-07 — [fps-watchdog], [ups], [top-kep], [viewer-alert]), and both media-attachment defects — Frigate snapshots now render and bambuddy finish photos are attached. ⚠️ server_name = chat.taild4189d.ts.net is permanent — changing it means wiping the database. ⚠️ The Admin Room must stay unencrypted or user management breaks forever. ⚠️ The mainframe cannot reach chat.taild4189d.ts.net (tailnet ACL blocks tag:telep → sidecars on 443) — test from a phone/laptop or against 127.0.0.2:8008. ⚠️ Never post test alerts into the live telep-ertesitesek room — use #relay-scratch:chat.taild4189d.ts.net. ⚠️ All 41 relay-sent events were redacted from the live room, so it looks empty — that is expected, not a delivery failure. ⚠️ Never mutate matrix-nio’s shared users_for_key_query from another thread — it kills the session mid-send. ⚠️ Continuwuity’s /rooms/{room}/event/{id} ignores the room in the path — trust the event’s own room_id field, never the URL you requested. 🔴 Tailscale SSH logs full command lines to the journal — never pass a secret as an argument (it lands there in plaintext), and remember your own grep command is in the journal too. ⚠️ Never run /etc/init.d/network reload|restart on telep-router, and bind 127.0.0.2, never 127.0.0.1.