Two new per-service tailnet nodes stood up on telep-mainframe this session: monitor (Netdata system monitoring, incl. the suspect-core alarm) and transmute (self-hosted file converter, CPU-pinned off the faulty core). Both follow the sidecar pattern and were later swept by the 127.0.0.2 hardening.
Netdata — the monitor node
URL
https://monitor.taild4189d.ts.net (tailnet only, no Funnel)
Named monitor, not netdata, so the hostname survives if the tool behind it is ever swapped. Superseded an earlier tailscale serve --https=8445 mount on the main node (now dead).
What it collects, and the suspect-core alarm
Per-core CPU (all 24 threads), package/core temps, RAPL power, memory, disk/NVMe, network, all Docker containers (state + health), systemd units, and lm-sensors board data. per cpu core utilization and package_throttle_count are off by default in netdata v2 and were explicitly enabled — without that there would be no per-core charts.
Where to look after a crash
coretemp labels map 1:1 onto core_id, so the faulty P-core is Core 16 in the Sensors chart and cpu8/cpu9 in per-core CPU. A core16.conf alarm file (/home/levander/netdata/config/health.d/core16.conf) watches Core 16 temp (warn >85, crit >95) and cpu8 thermal throttling — both currently CLEAR, both to: silent. Retention is 14 days at 1-second resolution (tier 0), so the minutes before an event are readable — but netdata cannot capture the final seconds of a hard panic (page cache is lost), which is why thermalwatch + netconsole stay in place. Read them together: netdata for the run-up, thermalwatch for the last seconds, netconsole for the panic text.
🔴 netdata notifies NOBODY — confirmed 2026-09-07
It is not just that the core16.conf alarms are to: silent: all 28 SEND_* methods are set to "NO", a deliberate override of the stock YES defaults. No netdata alarm can reach anyone by any channel. This dashboard is pull-only — you must go and look. Site-wide context (smartd also delivers nowhere, and no unit has OnFailure=): 2026-09-07-alert-source-inventory.
netdata's chart labels: matcher splits on whitespace
chart labels: label=Core 16 silently parses as two patterns (label=Core and 16) and the alarm template binds to nothing — no error is logged. Match on space-free labels instead: chart labels: chip_id=coretemp-isa-0000 feature=temp18.
GPU metrics — least-privilege NVML only
GPU metrics (RTX 3080: temp, util, power, clocks, VRAM) come via the compose deploy.resources.reservations.devices nvidia device-reservation form (matching the frigate house pattern), capped to NVIDIA_DRIVER_CAPABILITIES=utility — injects only nvidia-smi + NVML, not CUDA/graphics. Verified least-privilege: nvidia-smi present, libcuda* absent. The display stack was untouched (module refcount unchanged, no reload).
The GPU idles high — ~120 W at ~4% utilisation
A constant thermal load in the case worth a separate glance. Also: on a host driver update the netdata container must be recreated (docker compose up -d) so the toolkit re-injects matching driver libs, or nvidia-smi inside breaks until recreate.
Two deliberate gaps
Journal-in-browser is gated behind Netdata Cloud — unavailable here
The systemd-journal function is registered and the plugin genuinely reads the host journal, but the browser viewer returns 412 / requires Netdata Cloud SSO to raise the caller above anonymous-data. There is no agent-local override. Cloud is deliberately disabled (NETDATA_DISABLE_CLOUD=1, DO_NOT_TRACK=1, cloud.conf enabled=no; verified zero outbound sockets / DNS). So to read logs: ssh telep-mainframe journalctl -b -1 -e (the previous boot is the one that matters after a crash). Do not “fix” this by claiming the node to Cloud — that reverses the privacy decision.
monitor is an unauthenticated window onto the whole host
The container reads /proc, /sys, the full systemd journal, /etc/passwd, and the Docker socket (:ro), with pid: host and apparmor:unconfined (but notprivileged; only SYS_PTRACE + SYS_ADMIN). Netdata has no auth — anyone on the tailnet who reaches the node sees every process, container and hardware detail. Fine for a single-owner tailnet, but front it with the existing oauth2-proxy/pocket-id pair if that ever changes.
Transmute — the transmute node
Self-hosted file converter (images / video / documents / data), live at https://transmute.taild4189d.ts.net (tailnet only, no Funnel). Node transmute / 100.73.152.121, tag:telep, unit tailscaled-transmute.service. Image ghcr.io/transmute-app/transmute:latest, loopback 127.0.0.2:3313 (post-hardening). Data in the transmute_data named volume; no host mounts (upload-driven).
CPU-pinned off the faulty core — this is the whole point of the deployment
Video transcode via FFmpeg is exactly the sustained all-core load that triggers the MCE fault. The compose pins:
cpuset: "0-7,10-23" # CPUs 8 and 9 (the faulty P-core, core_id 16) excludedcpus: 6.0 # hard CFS quota — caps total draw to ~25% of the box
Both matter: the cpuset keeps work off the bad core, but an MCE rendezvous involves every online CPU, so the quota holds machine-wide load below the failure condition. Verified at Docker, host-cgroup and in-container levels — CPUs 8/9 appear in none. Cost: video transcode ~4× slower; image/doc/data conversions unaffected. The safe speed knob is raising cpus: — never widen the cpuset back over 8,9. The protection is per-container: a hand docker run or dropping the limits loses it.
Guest access + an on-disk admin password
Transmute has no auth-none mode. It ships a user system; guest access is enabled via ALLOW_UNAUTHENTICATED=true, so anyone on the tailnet clicks “Use as Guest” (one click, no login) and can upload arbitrary files that get fed to FFmpeg / Pillow / Pandoc (all with parser-CVE history). A bootstrap admin account levander exists with a generated password at /home/levander/transmute/.admin-password (chmod 600) — rotate or delete it if that credential is unwanted. Two settings were kept independent of auth choice: AUTH_SECRET_KEY in /home/levander/transmute/transmute.env (fixes a real bug where the JWT/session key was regenerated in memory every restart) and APP_URL=https://transmute.taild4189d.ts.net (correct external URLs behind serve).
OIDC was staged then reverted — true identity is deferred to tsidp
Pocket-ID OIDC was ~90% wired, then reverted; Transmute has no trusted-header auth, so the Drive’s whois→Remote-User pattern cannot be reused (the app never reads a request header for auth). Current posture is guest-gated-by-tailnet-ACL. The real identity path is tsidp (Tailscale-native OIDC), which was staged this session but not brought up.
Gotchas shared by both
Node approval: tagging at birth (--advertise-tags=tag:telep on tailscale up) satisfied device approval automatically — no owner click was needed for monitor; transmute printed the approval line but came up Running and routable. Always pass --hostname explicitly or up derives it from the OS hostname (telep-mainframe).
services.yaml has concurrent writers. Homepage tiles for both were added by byte-level surgical insertion (exact-line anchor, idempotent), never a YAML round-trip — the file holds Hungarian text + emoji group names that a reserialise would mangle. Re-read fresh immediately before editing.
ghcr.io from this host is flaky — the transmute image pull failed twice on i/o timeout (general packet loss to GitHub, not a blackholed IP); docker compose pull here may need retries.