Levandor

UPS shed/restore hardening — boot-time reconcile so a missed ONLINE can't leave services shed

4 min read

Hardened the NUT UPS outage automation on telep-mainframe so that services shed during a battery event are always brought back — even when the mains-restore (ONLINE) event never fires because the box was powered off when power returned. Root-caused a real overnight failure where kb-qdrant stayed down and knowledgebase.service crash-looped 2360 times.

For Agents

Shared restore logic: /usr/local/bin/power-restore.sh — idempotent, logs tagged [RECONCILE]. Called by BOTH: (1) the NUT ONLINE handler in /etc/nut/nut-outage-handler.sh, and (2) a new boot-time oneshot power-restore-reconcile.service (enabled, After docker + network-online). kb-qdrant now runs with --restart unless-stopped. knowledgebase.service gets a drop-in knowledgebase.service.d/wait-qdrant.conf that polls 6333/readyz (~60 s) before starting instead of crash-looping. Handler backup: /etc/nut/nut-outage-handler.sh.bak-*.

Root cause (the overnight failure)

The sequence, last night:

  1. Mains dropped → NUT ONBATT → outage handler shed non-essential services (KB, Qdrant, camwall, AirPlay, GPU/index jobs). Critically, kb-qdrant was explicitly docker stopped during the shed.
  2. Battery drained to LOWBATT → upsmon did a graceful systemctl poweroff. Correct behaviour.
  3. Mains returned WHILE THE BOX WAS OFF.
  4. BIOS “Restore on AC Power Loss” (which is ON — see telep-mainframe-handover §4) auto-powered-on the box → it booted.
  5. But the NUT ONLINE / restore event never fired — because the box was off during the mains-return transition, upsmon was never running to observe OB → OL. So the restore handler that restarts shed services never ran.
  6. Result: kb-qdrant (explicitly stopped during shed, with no boot recovery because it had no restart policy) stayed down. knowledgebase.service came up on boot, found no Qdrant at 127.0.0.1:6333, and crash-looped 2360× against the missing dependency.

The core design flaw

Restore was coupled only to the NUT ONLINE event. That event is unobservable if the box is powered off across the mains-return. A graceful poweroff + BIOS auto-power-on is a completely normal UPS outcome, so “power came back while off” is not an edge case — it’s the expected path for any outage that outlasts the battery.

Fix (all on the box)

1. Extract restore into a shared, idempotent script

Restore logic was pulled out of the inline NUT handler into /usr/local/bin/power-restore.sh:

  • Idempotent — safe to run whether services are already up or still shed; it reconciles to the desired “everything running” state rather than blindly issuing starts.
  • Logs are [RECONCILE]-tagged so boot-time reconciliation is distinguishable from a live ONLINE-triggered restore in the journal.

2. Call it from two places

  • The NUT ONLINE handler (/etc/nut/nut-outage-handler.sh) still calls it for the normal case (power returns while the box is up). Backup at /etc/nut/nut-outage-handler.sh.bak-*.
  • A new boot-time oneshot power-restore-reconcile.service (enabled, ordered After docker + network-online.target) calls it on every boot. So a missed ONLINE can never again leave services shed — the box reconciles itself into the running state as it comes up.

3. Give Qdrant its own boot recovery

kb-qdrant set to --restart unless-stopped so docker brings it back on boot on its own — belt-and-braces with the reconcile service. (unless-stopped still respects a deliberate manual docker stop.)

4. Stop KB crash-looping on a cold Qdrant

Drop-in knowledgebase.service.d/wait-qdrant.conf makes knowledgebase.service poll 6333/readyz (~60 s) before starting instead of failing instantly and letting systemd restart-loop it. KB now waits for its dependency to be ready rather than hammering a missing one 2360 times.

Gotcha — sudo tee silently dropped a shebang

While installing a script, sudo tee silently dropped the #!/usr/bin/env bash shebang line. On direct execution the kernel fell back to /bin/sh (dash), which choked on a bash array in the script.

Transfer shebang-bearing files as base64

When writing an executable script to the box through sudo tee / heredoc / SSH stdin, the first line (shebang) can be eaten. Use base64 transfer (base64 -d on the far side) for any file whose first line is load-bearing. Same class of quoting/first-line hazard seen elsewhere in this homelab’s remote-write workflows.

Result

  • A battery event that outlasts the UPS → graceful poweroff → BIOS auto-power-on → boot now self-heals: reconcile brings back every shed service, Qdrant is up (restart policy + reconcile), and KB waits for Qdrant instead of crash-looping.
  • The failure is no longer coupled to observing a NUT event that may never fire.

Graph View

Backlinks