Levandor

telep-mainframe "randomly turns off overnight" — Mains Power, Not the PC

6 min read

The i9-12900K / RTX 3080 telep-mainframe has repeatedly turned off by itself overnight and stayed dark until manually powered on. Root cause, conclusive: a brief mains power interruption, not a fault in the PC. The clincher is that the telep-router (separate hardware, separate PSU) died at the same instant — only a shared-mains event can take down both.

For Agents

Verdict: external mains power loss. The PC is healthy — no thermal, no PSU-load trip, no software shutdown. The desktop “stays off” only because its BIOS Restore on AC Power Loss is set to stay off (common default); the router auto-powers-on (embedded, no power button) so its recovery is invisible. Real fix: a UPS (with NUT logging). Free stopgap: flip the BIOS AC-loss policy to Power On — needs physical BIOS access, cannot be done over SSH.

The complaint

Recurring unclean shutdowns (“crash” in last -x): Jul 23 06:36, Jul 19 06:59, Jul 17 21:28, Jul 16, Jul 14. The PC turns off by itself and stays off — a boot only appears many hours later when someone presses the power button (e.g. 06:36 crash → 13:31 manual boot = ~7h gap), never an auto-restart.

Evidence chain (the reusable method)

This is the valuable part — a repeatable way to prove “mains, not PC”.

  1. The journal stops dead mid-normal-operation. journalctl -b -1 tail ends on a routine intruder-alarm arp-scan presence check at 06:36:10, then nothing. No systemd Stopping… sequence, no kernel panic/stack trace, no thermal warning, no OOM. That absence is diagnostic: a software shutdown always logs Stopping; a panic logs a trace. A clean dead-stop = power physically removed.
  2. Same signature across boots. All three inspected crash tails (boots -1/-2/-3) show the identical dead-stop, and all happened during low activity (just the alarm loop) — which argues against a load-induced PSU trip.
  3. Thermal / hardware fully ruled out. journalctl | grep -iE "thermal|mce|machine check|hardware error|throttl|Xid" across all boots = nothing. Current temps cool: acpitz 27 °C, x86_pkg_temp 36 °C, coretemp ~30 °C, GPU 47 °C / 128 W of 400 W. Overheating impossible.
  4. Software poweroff ruled out. root + levander crontabs empty; no systemd poweroff/suspend timers; atq empty; /sys/class/rtc/rtc0/wakealarm empty; sleep.target + suspend.target masked.
  5. The clincher — cross-check an independent always-on device. The telep-router (separate hardware, separate PSU) went down at the same instant: mainframe journal died 06:36:10 CEST; router booted 06:36:27 CEST (17 s later = its POST time after power returned). The router boot epoch was derived from date +%s minus /proc/uptime. A PSU fault in the PC cannot take down the router too — only a shared mains event can. This single cross-check is what turns “probably power” into “definitely mains”.

Dead-stop signature = power physically removed

A journal that stops dead with no Stopping…, no panic/stack trace, and no thermal/MCE line was not shut down by software or the OS — the power was physically cut. A software shutdown always logs a Stopping sequence; a kernel panic always logs a trace. Their absence is the diagnosis, not missing data.

To tell a PC PSU fault from a mains event, cross-check an independent always-on device

Derive the router’s (or any always-on box’s) boot time — date +%s minus the first field of /proc/uptime — and compare it to the instant the PC’s journal died. Same-timestamp death = mains (one shared event took down two independent PSUs). Only the PC died = its own PSU / hardware. One cross-check settles it.

Why only the PC appears to “stay off”

Not a different failure — a BIOS policy difference:

  • The router auto-powers-on when mains returns (embedded gear, no power button) → its recovery is invisible, it’s just back.
  • The PC’s BIOS Restore on AC Power Loss is set to STAY OFF (the common default) → the desktop sits dark until someone presses the power button.

Every last -x “crash” is followed by a boot only many hours later, never an auto-restart — consistent with the stay-off BIOS policy and inconsistent with a panic (which would reboot).

Current state / impact

  • Root ext4 fs state clean, no ext4 / I-O errors, 0 failed units.
  • All services recovered on the latest boot: camwall, wifi-usage, intruder-alarm, top4-web, docker, incus active.
  • Frigate runs as a docker container — frigate.service shows inactive because it is not a systemd unit, it is docker compose; the frigate + frigate-notify containers are healthy.

So journaling has survived the hard cuts so far. But repeated ungraceful power loss risks eventual ext4 / Frigate-recording corruption — this is luck, not resilience.

Recommendations (priority order)

  1. UPS — the real fix for unstable mains. A ~600–900 VA consumer UPS under PC + router + TV rides through brownouts / brief cuts. Get one with USB and wire up NUT so it (a) logs every transfer-to-battery / input-sag event (proving grid-outage vs overloaded-circuit) and (b) triggers a graceful shutdown on long outages.
  2. Free stopgap — flip the BIOS AC-loss policy. Set BIOS Restore on AC Power Loss → Power On / Last State so the PC auto-recovers like the router — converts “dead until I get home” into a ~90 s reboot. Requires physical BIOS access, cannot be done over SSH. Does not prevent the ungraceful cut.

Follow-up done: the disk now auto-unlocks on reboot (TPM), so a boot no longer stalls at a passphrase prompt

Root was LUKS2 passphrase-only, so even after the box powered back on it sat at an unlock prompt (unreachable over SSH). On 2026-07-24 the TPM was bound with clevis so the disk auto-unlocks unattended at boot, passphrase kept as fallback — 2026-07-24-luks-tpm-autounlock. This does not fix power recovery on its own: the BIOS Restore on AC Power Loss = Power On flip (#2) is still what makes the box come back at all after a mains cut; auto-unlock only removes the second barrier once it boots.

  1. Rule out an overloaded shared circuit. i9-12900K + RTX 3080 (up to ~500 W) + 4K LG TV on one circuit could be tripping a breaker rather than a grid outage. Mixed event times (06:36, 06:59, 21:28) fit either; the UPS logging in #1 distinguishes them.

Graph View

Backlinks