Levandor

Runbook: A homelab web service won't load

4 min read

Which path are you on?

  • Local (on-site, *.telep.lan) → DNS wildcard on telep-router → Caddy on telep-mainframe → backend. Works without internet.
  • Remote (away, *.taild4189d.ts.net) → Tailscale serve → backend. Needs internet. If the WAN is down, the remote path is dead — use the local *.telep.lan path instead.

If EVERYTHING by-domain is down at once (all *.telep.lan + admin/id.levandor.io) → it's Caddy

A single dead caddy.service takes down all domain access. Most common after a reboot: Caddy loses a :443 bind-race against the LAN IP coming up. Go straight to caddy-boot-race — systemctl status caddy, fix = ip_nonlocal_bind + systemctl start caddy.

Symptoms

  • A service name (e.g. files.telep.lan, frigate.telep.lan) won’t load / times out / shows 502.
  • Works locally but not remotely, or vice-versa.
  • All names dead at once (not just one) → caddy-boot-race.

Diagnose → Fix — LOCAL (*.telep.lan)

  1. DNS resolves to the mainframe? 
    nslookup <svc>.telep.lan          # must answer 192.168.1.123
    Not resolving → the router wildcard is missing/DNS is down → see no-wifi-or-dhcp.
  2. Caddy up and config valid? On the mainframe: 
    ssh levander@100.115.209.87
systemctl status caddy
caddy validate --config /etc/caddy/Caddyfile
ss -lntp | grep ':80 '            # Caddy listening on :80
  3. Backend up? Ask Caddy’s upstream directly by Host header — a 502 = backend down: 
    curl -s -o /dev/null -w '%{http_code}\n' -H "Host: <svc>.telep.lan" http://127.0.0.1/
    200 = healthy; 502 = fix/restart the backing service (its port is in the map below).

Name → backend map (the 502 tells you which port to check)

files→:8334 · frigate→https://:8971 · top→:8090 · print→:8095 · whiteboard→:8790 · ruview→:3000 · go2rtc→:1984 · kraken→192.168.1.200:8080 (the Pi). Full table + Caddy gotchas: 2026-08-15-telep-lan-split-horizon-caddy.

Diagnose → Fix — REMOTE (Tailscale)

  1. Node online? 
    tailscale status | grep <node>
sudo tailscale serve status
  2. Remember Tailscale needs internet. If the site WAN is down, no tailnet name resolves from outside — switch to the local *.telep.lan path (above) while on-site.
  3. Getting a 502 from a tailnet name? Then the serve mount is up and its backend is not. Two checks, in this order, both on the mainframe: 
    sudo tailscale --socket=/run/tailscale-<node>/tailscaled.sock serve status  # which target?
docker ps -a | grep -i Exited                                              # the -a MATTERS

restart: unless-stopped does NOT guarantee a container is running

If the containerd shim fails before the task is created, there is no task for Docker’s restart policy to act on — the container rests at Exited (128) / Error: failed to create shim task: ttrpc: closed / RestartCount: 0 and stays dead indefinitely. freecad-mcp sat like that for three days and 502’d the whole FreeCAD MCP. Fix is a plain docker start <name>. See 2026-09-05-freecad-mcp-502-dead-container-and-qt-event-loop-wedge. Never treat “it has a restart policy” as evidence that it is up.

Known specifics

kraken.telep.lan 502 = the Pi is offline, not a proxy bug

The Caddy entry proxies to 192.168.1.200:8080 (the KrakenSDR Pi). A 502 here means the Pi (192.168.1.200) is down — see host-offline and krakensdr-df. The proxy config is correct; no action unless the Pi stays down.

  • jellyfin.telep.lan was a guessed name with the backend down — flagged for removal/rename, not a real outage.
  • frigate speaks HTTPS with a self-signed cert (tls_insecure_skip_verify); ruview needs header_up Host {upstream_hostport} (421s otherwise). Details in the split-horizon note.

Deeper causes / links

Graph View

Backlinks