Levandor

telep-mainframe hard-crashes: MCE broadcast timeout on one physical P-core — a CPU hardware fault

11 min read

telep-mainframe hard-crashed four times in ~20 minutes on the night of 2026-08-30/31 after 8 days of uptime. Netconsole capture proves a CPU hardware fault: one physical P-core fails to enter the Machine Check Exception broadcast rendezvous under sustained all-core load, and the kernel panics. The embedding pipeline that “caused” it is only the load that touches the bad core, and the nvidia warnings everyone kept blaming are unrelated.

Verdict — hardware, not software

mce: CPUs not responding to MCE broadcast (may include false positives): 8-9 Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler

CPUs 8 and 9 are the two SMT threads of ONE physical P-core (core_id=16) on the i9-12900K. A single core failing the MCE rendezvous under load is the signature of a degrading/faulty core. No software change fixes this — the remaining actions are BIOS/firmware, core isolation, or RMA.

READ THIS BEFORE YOU BLAME NVIDIA

The log is full of WARNING … nv_drm_revoke_modeset_permission [nvidia_drm] (DKMS 550.163.01, Comm: vo = mpv, i.e. camwall). These are a red herring and are explicitly ruled out. They appear before and after the panic, and on healthy boots that never crash. Two separate agents independently concluded “nvidia_drm is crashing the host” and both were wrong and both retracted. The earlier note 2026-08-31-nvidia-drm-host-crash-embedding-pass is superseded by this one. Do not re-derive that dead end.

The evidence

Hosttelep-mainframe — i9-12900K, Gigabyte Z690 AORUS MASTER, BIOS F29 (2024-09-27)
Kernel6.12.100+deb13-amd64, tainted P/W/OE (proprietary nvidia module)
Panic (captured)2026-08-31 02:06:30 UTC (04:06:30 CEST)
Capture methodnetconsole → telep-router /tmp/netconsole/kmsg.log
Prior uptime8 days (2026-08-22 22:01 → 2026-08-31 03:50 CEST)

The panic

mce: CPUs not responding to MCE broadcast (may include false positives): 8-9
Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
Shutting down cpus with NMI
Kernel Offset: 0x3ae00000 from 0xffffffff81000000

The fault path

 <#MC>
 do_machine_check+0x930/0xab0
 mce_start+0x96/0x130
 mce_timed_out+0x70/0xb0
 mce_panic+0x165/0x280
 panic+0x192/0x2db
 kmsg_dump_desc+0x73/0xb0
 pstore_dump+0xfa/0x320
 efi_pstore_write+0x144/0x190 [efi_pstore]
 efivar_set_variable_locked+0xa3/0xf0
 efi_query_variable_store+0x1c3/0x210
 virt_efi_query_variable_info_nb+0x5d/0xe0
 arch_efi_call_virt_setup+0x18/0x80
 kernel_fpu_begin_mask+0xdb/0x100     <-- WARNING arch/x86/kernel/fpu/core.c:60
 </#MC>

Interrupted context was RIP: pick_task_fair+0x4e/0x130 (the scheduler) — incidental. An MCE is asynchronous; whatever was running when it fired is not the culprit.

Crash correlation

CrashBoot endedobsidian_sync active?
103:50:12 CESTyes — 03:49:12
203:54:17 CESTyes — 03:54:00
304:06:12 CESTyes — 04:06:09 ([5/192])
404:06:30 UTC (captured)yes — PID 82973 named in the panic

The trigger (not the cause)

CPU: 13 UID: 1000 PID: 82973 Comm: python

PID 82973 = obsidian_sync.sh → venv/bin/python obsidian_index.py — the kb-vectors embedding pipeline (torch 2.13.0+cpu, sentence-transformers). See 2026-08-31-obsidian-vault-qdrant-index-selection. It is a sustained all-core CPU load, which is exactly what exercises the faulty core.

Why the box looked stable for 8 days

The sync had been failing to start since 2026-08-26, so it did no real work. The box was idle-stable. New vault notes written on the night of 08-30/31 gave the indexer a genuine batch, and the machine died within seconds of it starting. The 8 days of uptime is not evidence of health — it is evidence that nothing loaded the bad core.

Mitigation applied: both obsidian-sync.timer and telep-kb-obsidian-sync.timer are disabled and stopped. This removes the trigger, not the fault. Any other sustained all-core job (a big compile, marker OCR, a full transcode) can reproduce it.

Confirmed: with both timers disabled the box has stayed up 43+ minutes with no further crash

Against a baseline of four crashes in ~20 minutes, this is good evidence that removing the trigger stops the panics.

It is mitigation, not a fix. The faulty core is untouched and still scheduled. Any sustained all-core load is expected to provoke the panic again — a long uptime here means “nothing has loaded core_id=16 hard”, exactly as it did during the 8 idle-stable days before the crash night. Do not read continued uptime as the problem being solved.

A systemd timer with OnBootSec= aimed at a host-crashing job is a BOOT LOOP

Host crashes → reboots → timer fires on boot → job runs → host crashes. Each cycle looks like a fresh spontaneous crash and the box never stays up long enough to diagnose. Disable the timer first, always. Also pinned: panic_on_oops=0, panic_on_warn=0, so the frequent nvidia WARNs can never themselves trigger a reboot.

06:20 recurrence — the box crashed AGAIN with the sync timers already disabled

Disabling the sync is NOT enough — ANY sustained all-core load triggers the fault

The box crashed again at 06:20 CEST, the same MCE on CPUs 8-9 (core_id=16), captured by netconsole — despite both obsidian sync timers being disabled since the first crash night. This time there was no obsidian indexer running. The trigger was the aggregate build load of the session’s own agents: concurrent Go builds (go build tsidp, go build tsauth-proxy), Docker image builds (the code-server OpenSCAD Dockerfile, transmute/netdata pulls), and multiple container recreates all landing at once.

Conclusion to carry forward: the earlier mitigation (“disable the sync”) removed one trigger, not the fault. The fault fires on any sustained all-core load, from any source. Until BIOS F34a + Secure Boot handling (or offlining core_id=16, or an RMA), the box must not run heavy load — that includes agent-driven builds, image builds, and batched container recreates, not just the embedding pipeline. This is why the session-added services pin their builds off the faulty core (taskset -c 0-15,17-23 for compiles, cpuset "0-7,10-23" for Transmute’s FFmpeg).

Everything auto-recovered on reboot: on-disk compose/unit/serve state meant every container came back (on 127.0.0.2 after the hardening sweep) and all verifications passed. A long uptime after this still only means nothing has loaded the bad core hard — read Why the box looked stable for 8 days again before reading uptime as health.

This recurrence also deferred tsidp (three bring-up attempts were defeated by it plus Aperture instability). Earlier same-night crashes (03:50 / 03:54 / 04:06) are the captured panic above.

Why every earlier crash was forensically invisible

/sys/fs/pstore was empty after every single panic, which is why months of crashes produced nothing to look at.

efi_pstore_write runs inside <#MC> context, where the FPU is unavailable. kernel_fpu_begin_mask trips the WARNING at arch/x86/kernel/fpu/core.c:60 and the EFI variable write fails. The crash dump is never persisted.

pstore will KEEP failing for MCE panics

This is structural, not a misconfiguration. Do not spend time “fixing pstore” for this failure mode. Netconsole is the only mechanism that works here. (Secondary, independent limit: efivarfs is 64% full — 160K/256K — so even a working write could be short on space.)

Thermal evidence — weak, and against a simple thermal story

Continuous sampling of the suspect core gives core_id=16 avg 38.1 °C — mid-pack, with other cores peaking higher. It is not the hot core.

The 54 °C reading was an artifact

An initial one-shot sensors read showed core_id=16 at 54 °C and briefly looked like a smoking gun. Continuous sampling showed that was a transient. One-shot thermal reads on a bursty box mean nothing — this is why thermalwatch samples continuously.

Instrumentation now in place

All three units below are confirmed enabled:

/usr/bin/ssh levander@telep-mainframe \
  'systemctl is-enabled netconsole-target.service thermalwatch.service drive-mcp.service'

1. netconsole-target.service — the only thing that captures the panic

telep-mainframe 192.168.1.123:6665  --(UDP, extended mode)-->  telep-router 192.168.1.1:6666

Receiver on telep-router: /usr/sbin/netconsole-rx + /etc/init.d/netconsole-rx (procd, enabled), writing /tmp/netconsole/kmsg.log.

/tmp on the router is RAM-backed — copy the log off after a crash

telep-router is OpenWrt; /tmp is tmpfs. A router reboot or power cut loses the panic log. Retrieve it before touching the router.

2. thermalwatch.service — the thermal run-up into a crash

/var/log/thermalwatch/samples.log — 10 s interval, fsync’d per line (so the last sample before a hard power loss survives), ~10 days of retention. Each line tags the suspect core explicitly as SUSPECT_core16_cpu8_9=, so a crash correlation is one grep away.

3. kernel.printk = 5 4 1 7

/etc/sysctl.d/99-crash-capture.conf.

The box boots quiet → console_loglevel 4 → every KERN_WARNING was being silently dropped

Including soft-lockup, hard-lockup and hung-task warnings. Months of “no kernel output at all before the crash” was partly this: the warnings existed, the console just never printed them. Raising console_loglevel to 5 is what made the pre-panic picture visible at all. If you ever re-image or edit the kernel cmdline, re-check this.

What to do after the next crash

Run these in order, from a workstation (use /usr/bin/ssh — the ssh shell alias is broken on the Mac). Do the router copy first: that log is in RAM.

# 1. PANIC LOG — copy off the router's RAM-backed tmpfs before anything else
/usr/bin/scp root@telep-router:/tmp/netconsole/kmsg.log \
  ~/crash-evidence/$(date +%F-%H%M)-kmsg.log
# fallback if MagicDNS is down:  root@100.69.112.32
 
# 2. Confirm it is the same fault (expect the MCE broadcast timeout + CPUs 8-9)
grep -nE 'MCE broadcast|Kernel panic|do_machine_check|CPUs not responding' \
  ~/crash-evidence/*-kmsg.log
 
# 3. THERMAL RUN-UP — the ~10 minutes before the crash
/usr/bin/ssh levander@telep-mainframe \
  'sudo -n tail -n 200 /var/log/thermalwatch/samples.log'
 
# 3b. just the suspect core's trace
/usr/bin/ssh levander@telep-mainframe \
  "sudo -n grep -o 'SUSPECT_core16_cpu8_9=[0-9.]*' /var/log/thermalwatch/samples.log | tail -n 200"
 
# 4. What was running (was it an all-core job again?)
/usr/bin/ssh levander@telep-mainframe \
  'journalctl --list-boots | tail -5; journalctl -b -1 -n 300 --no-pager'
 
# 5. Confirm BOTH ends are still live for the NEXT crash
/usr/bin/ssh root@telep-router '/etc/init.d/netconsole-rx status; ls -l /tmp/netconsole/'
/usr/bin/ssh levander@telep-mainframe \
  'systemctl is-enabled netconsole-target.service thermalwatch.service; systemctl is-active thermalwatch.service'

Do NOT bother with /sys/fs/pstore

It will be empty. See Why every earlier crash was forensically invisible.

Next steps — hardware, owner decision

Ordered cheapest-first.

  1. Reset BIOS to defaults. Clears XMP, any undervolt/OC. Free, and the single most likely cheap win.
  2. BIOS update F29 → F34a (Jul 2026). Its changelog specifically notes “Fixed VccSA option visibility for 12th Gen Intel Core processors” — VccSA is the System Agent rail, and its behaviour is plausibly relevant to a core failing an inter-core rendezvous.
  3. Stopgap to keep the box usable: offline or isolate CPUs 8-9 (core_id=16) — nosmt is too blunt; prefer isolcpus= / cpu0.. offlining or a kernel cmdline maxcpus variant that removes just that core’s threads.
  4. Check CPU warranty. A single core failing MCE rendezvous under load is consistent with degradation and is an RMA-able symptom.

BIOS F31/F32+ enable Secure Boot and pre-boot DMA protection BY DEFAULT — this WILL break camwall

The NVIDIA driver is an unsigned DKMS module. With Secure Boot on it will not load, the RTX 3080 exposes no display connectors, and the TV wall dies. After flashing anything ≥ F31, go back into the BIOS and disable Secure Boot before rebooting into Linux. Budget for the box coming back headless on the first boot.

Prefer Q-Flash Plus for the update

Q-Flash Plus flashes from a USB stick via the board’s own controller — no CPU and no OS involved. Given the whole problem is a faulty core, an in-OS or in-BIOS flash that hangs mid-write bricks the board. Use the dedicated Q-Flash Plus port + button.

This is NOT about microcode — don't update the BIOS expecting a microcode fix

Debian’s intel-microcode package already loads revision 0x3d at boot, which is newer than any microcode shipped in any of these BIOS versions. The BIOS microcode is overridden before userspace starts. Whatever a BIOS update buys here, it is board firmware / VRM / rail configuration — not CPU microcode.

Graph View

Backlinks