Levandor

telep-mainframe PSU-under-load resets (distinct from mains outages)

3 min read

telep-mainframe crash-rebooted ~3× in 30 minutes (04:11 / 04:20 / 04:32) while running many heavy jobs at once (GPU OCR + CPU embedding + kick-fix conversion + Frigate). This is a SECOND, distinct power problem from the overnight mains outages: the PSU cannot sustain the peak combined i9-12900K + RTX 3080 draw under full load. A UPS does NOT fix this one.

Two SEPARATE power problems on this box — don't conflate them

(a) Overnight MAINS outages → the wall power drops; the router dies WITH the PC → fix is a UPS. (b) PSU-can’t-sustain-peak-load resets → wall power is fine, the PSU browns out under load and the box hard-resets → fix is to serialize heavy jobs / cap threads, ultimately a bigger PSU. A UPS does nothing for (b). See 2026-07-23-mains-power-shutdowns for (a).

Diagnosis

  • NOT thermal: CPU 39 °C / GPU 47 °C at the time, zero thermal/MCE events in the journal.
  • NOT mains: the telep-router (separate PSU) stayed up 14h+ straight through the resets. Contrast the overnight outages, where the router died with the PC — that co-death is what makes those genuine mains events.
  • Dead-stop journal signature: the journal stops mid-operation, no clean Stopping shutdown sequence — the same “power physically removed” fingerprint as a mains cut, but here the power was removed by the PSU browning out, not the wall.
  • Reproducible trigger: the default 16-thread CPU embedding (all cores maxed, from 2026-07-24-kb-vectorize-complete) reset the box within 2–6 minutes every time.

The router-uptime cross-check is the discriminator between PSU and mains

Derive the router’s uptime (it’s on a separate PSU and always on). If the router died at the same instant as the PC → mains (fix: UPS). If the router stayed up while only the PC reset → the PC’s own PSU browning out under load (fix: serialize / bigger PSU). Same dead-stop journal signature either way — the independent always-on device is what tells the two apart.

Mitigation that worked

Cap CPU threads to 4 + nice, and serialize heavy jobs

OMP_NUM_THREADS=4 / torch.set_num_threads(4) + nice on the embedding job, and run heavy jobs one at a time (not GPU OCR + CPU embedding + kick-fix conversion + Frigate simultaneously). With the cap the box stayed up 45+ minutes and completed the full 2839-chunk index.

  • Real hardware fix: a higher-wattage / higher-quality PSU that can hold the peak combined i9-12900K + RTX 3080 draw. The thread cap is a workaround, not a cure.
  • Operationally: treat the box as single-heavy-job-at-a-time until the PSU is replaced. The OCR queue was already GPU-serialised (single worker); the new discipline extends “one at a time” across CPU-heavy work too.

Graph View

Backlinks