Runbook: The mainframe or the Pi dropped off

Mid-incident? Recovery access (don't rely on the LAN/MagicDNS)

ssh root@100.69.112.32          # router  (telep-router, over Tailscale)
ssh levander@100.115.209.87     # mainframe (telep-mainframe, over Tailscale)

Don’t confuse the two LAN hosts:

• Mainframe = 192.168.1.123 (enp5s0), mac d8:5e:d3:a7:05:d6
• Raspberry Pi (KrakenSDR rig / krakensdr-df) = 192.168.1.200

Symptoms

• The mainframe (telep-mainframe) or the Pi fell off Tailscale / stopped responding.
• A dependent service is 502-ing because its host is gone (e.g. kraken.telep.lan → the Pi).

Diagnose → Fix

1. Can you reach it over Tailscale? If the tailnet IP responds, the host is up and it’s a service problem, not a host-down — go to service-unreachable.

   ssh levander@100.115.209.87     # mainframe
   ssh levander@raspi.local        # Pi (or via its tailnet node kraken-rig)

2. Most common cause: it couldn’t get/renew a DHCP lease — so it lost its LAN IP and fell off Tailscale with it. Check from the router:

   ssh root@100.69.112.32
   grep -iE '192\.168\.1\.123|192\.168\.1\.200' /tmp/dhcp.leases
   ip neigh | grep -iE '192\.168\.1\.123|192\.168\.1\.200'

   No lease / no neighbour entry → DHCP is the problem, not the host. Fix DHCP and the host re-leases within ~1 minute → no-wifi-or-dhcp.
3. DHCP is healthy but the host is still gone → it’s genuinely down (power, crash, or unplugged).
   • Mainframe: [!tip] BIOS “Restore on AC Power Loss” is ON — after a mains outage it auto-powers-on. If it’s off after an outage, wait for boot, or physically check the PSU switch. It self-recovers its full stack on boot (UPS shed/restore is hardened).
   • Pi: check its USB-C power and eth0 link; it has no auto-power BIOS setting.

A mains outage can leave the mainframe network-dead even after it boots

On 2026-08-15 the box rebooted, then sat network-dead for ~8.5 h because DHCP was down on the router (a duplicate reservation). The host was fine — the network wasn’t. Always rule out DHCP/DNS (step 2) before assuming the host itself failed.

Deeper causes / links

• DHCP/DNS down site-wide (the usual reason a host “drops off”): no-wifi-or-dhcp · 2026-08-15-dhcp-outage-duplicate-reservation-postmortem.
• Mainframe power/UPS behaviour (shed/restore, BIOS auto-power-on, self-recovery on boot): 2. Power & UPS resilience · 2026-08-08-ups-shed-restore-hardening.
• The Pi specifically (KrakenSDR rig, 192.168.1.200, shared with wifi-hunter): krakensdr-df · 2026-08-12-krakensdr-doa-rig.
• A service is down but the host is up → service-unreachable.

Related

• telep-mainframe · telep-router
• no-wifi-or-dhcp · service-unreachable · krakensdr-df
• runbooks-index — all homelab runbooks