Levandor

Homelab Runbooks — Index

4 min read

Action-first “when X breaks, do Y” guides for the homelab. Each runbook is symptoms → numbered diagnosis/fix with copy-paste commands → deeper causes with links to the detailed incident notes. Optimized for someone mid-incident.

Recovery access when the LAN is down

The LAN/DNS may be dead — reach the gear directly over Tailscale (raw IPs, not MagicDNS):

Don’t confuse the LAN hosts: mainframe = 192.168.1.123 (enp5s0, mac d8:5e:d3:a7:05:d6); Raspberry Pi (KrakenSDR rig) = 192.168.1.200; router = 192.168.1.1; cameras (telep-cc WiFi) = 192.168.30.x.

Symptom → Runbook

If this is happening…Go to
Can’t join WiFi / no IP / whole network down / internet gone for everyoneno-wifi-or-dhcp
Mac can’t see the 5 GHz SSID (telep1-5G) in the Wi-Fi menu but phones can2026-08-26-router-reflash-mac-5ghz-dfs-and-bambu-offline
Bambu H2S 3D printer offline / Bambu Studio IOTC … ErrCode -122026-08-26-router-reflash-mac-5ghz-dfs-and-bambu-offline
Router reflashed/reset — SSH host-key changed, key auth fails, demands root password2026-08-26-router-reflash-mac-5ghz-dfs-and-bambu-offline
A host web service won’t load / 502 / *.telep.lan or Tailscale name deadservice-unreachable
EVERYTHING by-domain down after a reboot — all *.telep.lan and admin/id.levandor.io — but backends are upcaddy-boot-race
The mainframe or the Pi dropped off Tailscale / stopped respondinghost-offline
AirPlay to the TV (telep-tv) — not appearing, won’t cast, no video, no audioairplay-telep-tv
Camera wall is black / not on the TV / a pane is frozencamwall-not-on-tv
All Frigate feeds dead + camwall stuck activating (start-pre) after a wifi reload — ifstatus cams = DEVICE_CLAIM_FAILED, ifup cams won’t fix it (needs /etc/init.d/network restart)camwall-not-on-tv · 2026-08-31-cams-vlan-device-claim-failed-wifi-reload
Camera alerts not firing — no Telegram person/car alerts (“Dezsi az őr” silent), logs flooded with Already notified on this zonecamera-alerts-not-firing
KrakenSDR DF — overdrive, no signal, ghost bearings, UI won’t startkrakensdr-df
Admin portal (admin.levandor.io) — passkey login 500/invalid_request, 502, or ssh actions failadmin-portal-login-broken
Disk won’t unlock / box powered itself off / need to swap-or-add a YubiKeyyubikey-luks-lockout
Nextcloud “Drive” (drive.taild4189d.ts.net) or code-server (code.taild4189d.ts.net) — won’t load, no SSO login, OnlyOffice editor blank, a Taildrive file is read-only in the web UI, /windows missing or writable, a file opens in the wrong app, or you need the occ / restart / rollback commands2026-08-31-nextcloud-drive-code-server-runbook
An MCP endpoint on the box returns 421 Invalid Host header over the tailnet while loopback works2026-08-31-nextcloud-drive-code-server-runbook · 2026-08-31-telep-kb-mcp-server
FreeCAD MCP tools all fail with Failed to get RPC status: [Errno 111] Connection refused — and nc -z cad…9875 says the port is OPEN2026-09-02-freecad-mcp-rpc-refused-gui-not-running
Mainframe hard-crashed / rebooted itself with no warning — especially during a heavy all-core job (embedding pass, build, transcode)2026-08-31-telep-mainframe-mce-hardware-fault · host-offline
/sys/fs/pstore is empty after a crash and there is no kernel log to look at2026-08-31-telep-mainframe-mce-hardware-fault

Two footguns that cause big outages

  • Never add a dhcp-host reservation without checking first (uci show dhcp | grep -iE '<ip>|<mac>'). A duplicate makes dnsmasq refuse to start → DHCP and DNS dead site-wide. See no-wifi-or-dhcp.
  • The AirPlay netifd patch must be reapplied after any OpenWrt sysupgrade, or wired→WiFi mDNS/AirPlay breaks again. See airplay-telep-tv.
  • Never run ykman config … on the mainframe without disarming the YubiKey live-lock first — re-enumerating the key reads as a removal and powers the box off mid-command. See yubikey-luks-lockout.

telep-mainframe has an unfixed CPU hardware fault — heavy jobs can panic it

One physical P-core (core_id=16, threads 8-9) fails the MCE broadcast rendezvous under sustained all-core load. Before running anything heavy on the mainframe, know that it may take the box down, and disable any OnBootSec= timer pointed at the job first or you get a boot loop. Panic capture is via netconsole on telep-router (/tmp/netconsole/kmsg.log, RAM-backed — copy it off first); /sys/fs/pstore is always empty for this fault. Full detail + post-crash commands: 2026-08-31-telep-mainframe-mce-hardware-fault.

Graph View

Backlinks