Levandor

Runbook: ALL .telep.lan / levandor.io down after a reboot (Caddy :443 boot-race)

3 min read

Every *.telep.lan service and both levandor.io portals (admin, id) are unreachable by domain after a reboot — but the backends themselves are fine. Cause: Caddy started at boot before the LAN IP 192.168.1.123 was assigned, failed to bind :443, and did not retry, so it stayed dead. Discovered 2026-08-17 after it had been down ~1.6 days (since the YubiKey-FDE reboots on 2026-08-15). Full incident: Caddy boot-race outage (2026-08-17).

This is a single-service, site-wide outage

One dead host service (caddy.service on telep-mainframe) takes down all domain-based access at once — home.telep.lan, files.telep.lan, frigate.telep.lan, admin.levandor.io, id.levandor.io, everything. It fails silently: nothing alarms, and it is easy to miss if all your work is over ssh / the console (which is exactly how the YubiKey reboots were done). The home.telep.lan dashboard’s tailnet path (telep-mainframe.taild4189d.ts.net:8450) still works because it bypasses Caddy — 2026-08-17-home-dashboard.

Symptoms

  • Nothing on *.telep.lan loads by name; both admin.levandor.io and id.levandor.io are dead.
  • DNS still resolves (nslookup files.telep.lan → 192.168.1.123) and the backends are up if hit directly.
  • Just after a reboot (especially the YubiKey-gated boots — see 2026-08-15-yubikey-gated-luks-fde).

Diagnose

On the mainframe (ssh levander@100.115.209.87 if the LAN name is dead):

systemctl status caddy          # look for: failed (Result: exit-code)
journalctl -u caddy -b | tail   # THE tell:
#   listening on 192.168.1.123:443: bind: cannot assign requested address

That bind error = Caddy came up before enp5s0 had 192.168.1.123. The bind 192.168.1.123 on the :443 blocks (needed so it doesn’t collide with Tailscale’s 100.115.209.87:443, see 2026-08-15-admin-portal-passkey-olivetin) is what makes Caddy sensitive to the IP not being up yet.

Fix (immediate)

sudo systemctl start caddy
# verify:
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: home.telep.lan' http://127.0.0.1/        # 200
curl -s -o /dev/null -w '%{http_code}\n' https://admin.levandor.io/ -k                        # 302

Also check oauth2-proxy — admin.levandor.io can be 502 even with Caddy up

During this incident the oauth2-proxy container was also down, so admin.levandor.io returned 502 after Caddy came back. cd /home/levander/admin-portal && docker compose up -d oauth2-proxy (or --force-recreate). See 502-after-auth.

Permanent fix (already applied 2026-08-17)

Two layers so this can never recur:

  1. net.ipv4.ip_nonlocal_bind=1 — the real fix for the race; lets Caddy bind an address before it is assigned to the interface. Persisted: 
    # /etc/sysctl.d/99-caddy-nonlocal-bind.conf
net.ipv4.ip_nonlocal_bind = 1
  2. systemd drop-in /etc/systemd/system/caddy.service.d/resilience.conf — belt-and-suspenders retry + ordering: 
    [Unit]
Wants=network-online.target
After=network-online.target
 
[Service]
Restart=on-failure
RestartSec=5

Quick mental model

All .telep.lan / levandor.io down after a reboot → systemctl status caddy for the :443 bind race. Fix = ip_nonlocal_bind=1 (already persisted) + systemctl start caddy; if still 502, also bring up oauth2-proxy.

Graph View

Backlinks