Levandor

telep-mainframe hard-crashes under the embedding pass — nvidia_drm trigger + two operational traps

4 min read

SUPERSEDED — the nvidia_drm conclusion on this page is WRONG

Netconsole capture later the same night proved the crashes are a CPU hardware fault: mce: CPUs not responding to MCE broadcast: 8-9 → Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler. CPUs 8-9 are the two SMT threads of one physical P-core (core_id=16). The nv_drm_revoke_modeset_permission warnings appear before and after the panic and on healthy boots — they are unrelated noise. Two agents independently blamed nvidia; both retracted. Read 2026-08-31-telep-mainframe-mce-hardware-fault instead.

This page is kept only for Trap 1 and Trap 2 at the bottom, which remain valid and generally useful.

The full vault embedding pass (2026-08-31-obsidian-vault-qdrant-index-selection) hard-crashes telep-mainframe. The kernel trace points at the nvidia DRM driver, not at the job — the job is a trigger, not the caller. Two operational traps found while diagnosing it are at the bottom; both are more generally useful than the crash itself.

Symptoms

  • Host hard-crashes (not OOM, not a clean panic-and-reboot) during the full embedding pass
  • 3 crashes in 3 attempts — the correlation with the job is total
  • The last crash logged no kernel output at all
  • Result: the notes index build is stuck at 27 points across 5 of 218 notes

Kernel trace

nv_drm_revoke_modeset_permission+0x327/0x340 [nvidia_drm]
  ← drm_file_free
  ← drm_release
Driver550.163.01
GPURTX 3080
Kernel paramnvidia_drm modeset=Y
DRM fd holdersXorg, mpv

Root cause assessment

The driver bug is pre-existing. The identical trace appears in logs from 2026-08-22, before any of this work.

But the embedding job is CPU-only torch (per 2026-07-24-kb-vectorize-complete the embeddings deliberately avoid CUDA) and never opens a DRM fd — it cannot be the code path in the trace. It is a trigger: sustained load perturbs timing around a DRM release path that Xorg/mpv are already exercising.

Real fix is either:

  • a driver update off 550.163.01, or
  • dropping nvidia-drm.modeset=1

Neither is free: driver 550 is load-bearing for Frigate (Running it on a CUDA GPU box (telep-mainframe) — 550 caps CUDA at 12.4 and the OCR venv is pinned to that), and per TV wall (HDMI) nvidia-drm modeset=1 is mandatory or the 3080 exposes zero display connectors — dropping it kills the TV wall.

Distinct from the 12V connector and PSU load hard-offs: those were electrical and are resolved; this one leaves a kernel trace.

Trap 1 — a systemd timer with OnBootSec= pointed at a host-crashing job is a BOOT LOOP

Host crashes → reboots → timer fires on boot → job runs → host crashes. The box never stays up long enough to diagnose, and each cycle looks like a fresh spontaneous crash.

Disable the timer FIRST when diagnosing anything that might take the host down. telep-kb-obsidian-sync.timer is currently disabled for exactly this reason — re-enabling it before the driver issue is fixed re-arms the loop.

Trap 2 — pkill -f '<pattern>' over Tailscale SSH kills your own session

On a Tailscale SSH host, pkill -f matches against full command strings — including the tailscaled SSH wrapper’s own command string, which contains the command you typed. So pkill -f 'obsidian_index' matches the wrapper running your pkill -f 'obsidian_index', and you kill your own session mid-command.

Use a bracketed pattern (pkill -f '[o]bsidian_index') or match on comm (pkill -x python3) instead. The failure looks like the host died — indistinguishable from the crash you’re chasing, which is what makes it dangerous here specifically.

Graph View

Backlinks