Levandor

Power Root Cause CONFIRMED (NVMe damage) + UPS decision + KB migration parked — resume 2026-08-07

13 min read

Session status/handover for telep-mainframe. Two threads: (1) the recurring “turns off by itself” is definitively confirmed as mains power loss — and the hard cuts are now actively damaging the NVMe SSD (SMART proof below); the fix is a UPS arriving ~2026-08-07. (2) The knowledgebase migration (hybrid search LIVE, Astro+Starlight rebuild STAGED) is parked mid-flight and fully resumable once power is stable. Read this to resume cleanly on/after Friday 2026-08-07.

2026-08-06 UPDATE — read 6. 2026-08-06 UPDATE — KB fixkick expansion LIVE, search-index PARKED, brownout root-cause upgraded first

The KB Astro site is now LIVE with a large fixkick.com expansion (135 → 388 EN pages), but the new pages’ search indexing is parked (~90/253 embedded) after another overnight outage. That outage also confirmed heavy CPU triggers the brownouts (§6c). Canonical builder is build_content.py, NOT convert.py (§6a). Resume steps for the embed are in §6b.

2026-08-06 RESOLVED — the load-triggered cuts were a missing GPU 12V connector, NOT marginal mains

The recurring under-load hard-offs are root-caused and fixed: the RTX 3080 had an unplugged 12V PCIe power connector (3 separate PSU cables, one bank disconnected), reconnected during reassembly. A graduated CPU+GPU stress test (up to 250 W CPU / 450 W GPU cap) then PASSED with zero crashes — so the §6c “heavy CPU triggers the brownouts on marginal mains” theory is superseded for the load-triggered resets. Full write-up: 2026-08-06-power-root-cause-gpu-12v-connector-stress-test-pass. A UPS + NUT is still worth installing for genuine mains events, and the deferred fsck + NVMe self-test still apply given the accumulated media_errors.

Power root cause was originally read as CONFIRMED-MAINS damaging the SSD — see the 2026-08-06 RESOLVED note above

This section captured the state on 2026-08-05: the box hard-cut ~6× in ~50 min (boots as short as 90 s). NVMe SMART showed 240 unsafe_shutdowns, 1440 power_cycles, and 1617 media_errors — power-loss-during-write actively corrupting the drive. The unsafe shutdowns are real and happened; their CAUSE is now identified as the GPU 12V connector (above), not marginal mains. See 1. Mains power — root cause CONFIRMED, and it is damaging the SSD and the gotcha at 5. Reusable gotcha.

1. Mains power — root cause CONFIRMED, and it is damaging the SSD

telep-mainframe (i9-12900K + RTX 3080) hard-cut ~6 times in ~50 minutes on 2026-08-05, with boots as short as 90 s. last shows every boot back to Jul 31 ended in crash — not a single clean shutdown. This upgrades the earlier mains diagnosis from “conclusive by router cross-check” to “confirmed with SMART hardware evidence, and now showing measurable drive damage.”

Ruled out: NOT thermal, NOT hardware

  • Temps cool: CPU (x86_pkg) 49 °C, GPU (RTX 3080) 55 °C, ambient 27 °C. Zero CPU thermal throttling.
  • No MCE / EDAC / kernel panic / oops in any boot’s journal.
  • Crash signature = normal logs then INSTANT silence, no warning. A thermal trip or PSU over-current protection shutdown would log first; the abrupt dead-stop with nothing preceding it = external power physically removed. (Same dead-stop method as the July 23 note.)

The proof — NVMe SMART

The clincher this session is nvme smart-log:

metricvaluemeaning
unsafe_shutdowns240240 hard power-offs — confirms mains, not software
power_cycles1440corroborates a long history of hard cuts
media_errors1617power-loss-during-write is actively corrupting/damaging the drive
percentage_used4%wear is low — drive is NOT failing yet
critical_warning0no imminent-failure flag

The 240 unsafe shutdowns are hardware-level proof of repeated mains cuts. The 1617 media errors are the new, worrying part: each power loss mid-write accumulates errors. The drive is not failing yet (4% used, no critical warning) but it is bleeding errors that will not stop until the ungraceful cuts stop.

The nvidia-drm nv_drm_revoke_modeset_permission WARNING is a red herring — do NOT chase it

The recurring kernel WARNING … nvidia-drm-drv.c:1220 nv_drm_revoke_modeset_permission is a known-benign cosmetic nvidia driver warning that fires on Xorg/modeset start. It is unrelated to the crashes. Noted here so a future session does not waste time treating it as the cause.

TODO once power is stable

  • Run fsck on the root fs and an NVMe self-test (nvme device-self-test), given the 1617 media errors — verify no silent filesystem corruption accumulated.
  • Consider an electrician for the site circuit/wiring. A UPS treats the symptom (rides through the cut) but does not fix bad wiring / a failing site circuit — if brown-outs persist even on the UPS’s AVR, the wiring is suspect.

2. FIX / DECISION: UPS (arriving ~2026-08-07)

The real fix. Requirements for this box (RTX 3080 + active-PFC PSU):

  • Pure sine wave — MANDATORY. Active-PFC PSUs can shut off or behave erratically on the stepped/simulated sine wave of cheap UPSes.
  • AVR (Automatic Voltage Regulation) — for the brown-outs (corrects sags without switching to battery).
  • ~1500 VA / 900–1000 W — the system peaks ~550–650 W under combined CPU+GPU load; leaves headroom.
  • USB + NUT auto-shutdown — so the OS shuts down cleanly on a long outage (this is what stops the unsafe-shutdown / media-error bleed).

Recommended: CyberPower CP1500PFCLCD (PFC Sinewave) or APC Smart-UPS 1500. Go online / double-conversion (CyberPower OL series / Eaton 5PX / APC SRT) only if brown-outs persist even on line-interactive AVR.

PLAN when the UPS arrives (2026-08-07)

Install NUT (Network UPS Tools) on the Debian box and wire it to the UPS over USB for clean auto-shutdown on long outages, plus event logging (transfer-to-battery / input-sag). This is the step that actually stops the NVMe media-error accumulation. Until then, the box keeps taking ungraceful cuts.

3. KB migration — PARKED, fully staged, resume 2026-08-07

Two pieces of the knowledgebase overhaul. Piece A is live; piece B is staged but not cut over.

3a. Hybrid search — DONE + LIVE (on current mkdocs KB)

  • New /home/levander/knowledgebase/hybrid.py: a rank_bm25 keyword index (heading ×3 / manual ×2 boost) fused with the existing Qdrant semantic search via Reciprocal Rank Fusion (k=60).
  • Wired into /api/search — backward-compatible; added rrf / match fields to results.
  • Legacy lunr /browse search retired — unified to one search across the site.
  • Backups on the box: app.py.bak-hybrid, searchui.py.bak-hybrid, mkdocs.yml.bak-hybrid.
  • Verified the exact-title fix: e.g. “78,000 Miles” moved #3 → #1; code query “61a40” now #1.

3b. Astro + Starlight rebuild — STAGED (not cut over)

Staged at /home/levander/kb-astro/ — Astro 5 + Starlight 0.36, built with Bun (no Node).

  • Full content converted via convert.py: ~2203 pages (EN ~1211 + HU ~992), all 6 folders (suzuki-vitara, suzuki-sidekick, chevy-tracker, kick-fix, kickfix-docs, consolidated), 68 section landing/index pages (replicating gen_index.py folder/manual catalogs), HU /hu/ locale, splash page.
  • Directory-URL scheme preserved — byte-identical to the mkdocs URLs (critical for not breaking links).
  • Image optimization ON (default sharp → webp / lazy / responsive). The build was at ~1455 / 5124 images in the persistent node_modules/.astro cache when parked = RESUMABLE from the cache.
  • src/components/Search.astro island calls /api/search (the same hybrid endpoint from 3a).
  • Wiring diagrams DROPPED per user decision — the AI-OCR wiring graphs were inaccurate. Scanned wiring images stay as normal page images (this supersedes the wiring-extraction work; see 2026-07-30-kb-wiring-gallery-pivot-complete).

For Agents — RESUME STEPS (only once power is stable)

  1. Finish the build (resumes from the webp cache): 
    cd ~/kb-astro && PUBLIC_KB_API= ~/.bun/bin/bun run build
    Run it detached so a power cut does not kill your shell: 
    systemd-run --unit=kb-astro-build cd ~/kb-astro && PUBLIC_KB_API= ~/.bun/bin/bun run build
  2. VERIFY complete: all 6 folders populated; chevy-tracker deep page + landing return 200; HU pages 200; images resolve.
  3. Atomic cutover: 
    cp -a knowledgebase/app.py app.py.bak-astro
cp -a ~/kb-astro/dist ~/knowledgebase/site-astro
mv ~/knowledgebase/site ~/knowledgebase/site.mkdocs-bak
mv ~/knowledgebase/site-astro ~/knowledgebase/site
    Then edit app.py so Flask / and /search no longer serve the old SEARCH_HTML — let the catch-all serve Astro’s site/index.html. Keep /api/search, /upload, /hu, /browse, /semantics, /drafts, /generate, /lang.
  4. Restart knowledgebase.service; verify live on https://knowledgebase.taild4189d.ts.net/.

ROLLBACK (if the Astro cutover is bad) 

mv ~/knowledgebase/site ~/knowledgebase/site.astro-bad
mv ~/knowledgebase/site.mkdocs-bak ~/knowledgebase/site
cp -a app.py.bak-astro knowledgebase/app.py

Then restart knowledgebase.service.

Serving model chosen: Flask keeps serving; only the site/ directory swaps mkdocs → astro. Same URL, and /upload OCR ingest + /hu + /api are all preserved.

4. Other parked items

RuView (WiFi-CSI through-wall sensing) — STOPPED, blocked on hardware UNBLOCKED 2026-08-11

Superseded 2026-08-11 — real ESP32-S3 CSI is now flowing end-to-end

An ESP32-S3 was bought, flashed and provisioned: the container was recreated with -p 5005:5005/udp (the ingest port had only ever been EXPOSEd, never published — nothing could reach it), switched to CSI_SOURCE=esp32, given a persistent /app/data volume, and moved to its own tailnet node https://ruview.taild4189d.ts.net. Presence/motion/RSSI/variance are real; person-counting is not trustworthy and --calibrate has never been run. Full write-up, including the firmware trap that silently zeroes CSI yield on a display-less S3: 2026-08-11-ruview-esp32-csi-real-hardware.

  • github ruvnet/ruview. Backend + UI stood up on the mainframe as docker container ruview (image ruvnet/wifi-densepose, ~43 MB, simulated mode) at https://telep-mainframe.taild4189d.ts.net:8448/ui/index.html. — both the simulated mode and the :8448 URL are now obsolete.
  • Currently STOPPED — docker start ruview to resume. The container was recreated on 2026-08-11 (safe: it had no volumes, only a regenerable 16 K session-secret).
  • Honest assessment (still broadly holds): the backend / CSI plumbing is real, but the “densepose / heartbeat through walls” ML is largely heuristic/stub per the authors’ own README. Confirmed with real hardware — the pretrained model is 48 KB / 2048 LoRA params / 12 min of training, and its “82.3%” is a temporal-triplet representation metric, not presence accuracy.
  • Blocked on hardware No longer blocked. ESP32-S3 DevKitC-1 flashed and deployed at yield=33–37 pps. (The nRF genuinely has no WiFi CSI and remains unusable for this.) The ESP-IDF v5.3.1 + espressif/esp-csi fallback install at ~/esp/ went unused — the working build path is Docker espressif/idf:v5.4 against the repo’s own firmware/esp32-csi-node.

5. Reusable gotcha

Do NOT run heavy / write-heavy builds on telep-mainframe until the UPS is installed

Power cuts during writes add NVMe media errors (already at 1617 — see 1. Mains power — root cause CONFIRMED, and it is damaging the SSD). Mitigation used this session: resumable builds via an on-disk cache (the Astro webp cache) + atomic, rollback-able cutovers (swap the site/ dir, keep a .bak). If you must build before 2026-08-07, use those patterns so a cut costs minutes, not the whole job.

6. 2026-08-06 UPDATE — KB fixkick expansion LIVE, search-index PARKED, brownout root-cause upgraded

Session delta added 2026-08-06. The KB overhaul moved on substantially: a large fixkick.com content expansion is built and LIVE (browse/render), but the new pages’ search indexing is parked (~90 of 253 embedded) because the box died again overnight. The overnight outage also upgraded the power finding to a confirmed cause-and-effect: heavy CPU triggers the brownouts. All of this is resumable once the UPS is in (still due Fri 2026-08-07).

6a. KB fixkick expansion — DONE + LIVE

The knowledgebase (Astro + Starlight, knowledgebase.taild4189d.ts.net) was expanded from fixkick.com — the live successor to the archived kick-fix.com — growing the EN corpus 135 → 388 pages.

  • Added ~253 pages, incl. 23 TSB bulletins under kick-fix/tsb/, plus Wayback-Machine-recovered dead pages (e.g. schematics-run, which 500s on live fixkick but had a 2017 snapshot), and three new sections: general, power-elect, tsb.
  • 920 internal .html cross-links rewritten → 0 broken internal .html remain; 344 links dropped (dead / forum-junk). Driven by kickfix_linkmap.json + ADDED_MANIFEST.json. Backup: manuals-src/docs/kick-fix.bak-linkrewrite.
  • Single-child folders flattened — the folder URL serves its one page directly with a plain sidebar link and a child→folder redirect: good-bad-ugly, data, trailer-wiring, + several kickfix-docs folders. Logic lives in ~/kb-astro/build_content.py; flatten_redirects.json is wired into astro.config redirects.
  • Deterministic sidebar-focus deployed — an astro.config head script scrolls .sidebar-pane to the active item on astro:page-load + astro:after-swap + double-rAF + timeout + MutationObserver, to beat Starlight’s scroll-restore race. ⚠ The live Playwright click-through proof was NOT yet run (box went down) — verify on resume.
  • Glossary / acronym search expansion LIVE: hybrid.py does query-expansion from kb-vectors/glossary.json — 325 bidirectional entries (154 from the Vitara manual abbreviation tables + 171 from the fixkick JARGON page): ABS ↔ anti-lock brake system, EGR, TPS, “kick” → Sidekick/Tracker, etc. Exact matches still rank first.
  • Qdrant/BM25 target after a full embed ≈ existing 3166 + new-page chunks.

BUILD TOOL GOTCHA — the canonical builder is build_content.py, NOT convert.py

The only correct builder for the real KB build is ~/kb-astro/build_content.py — a full regen: it rebuilds the section indexes and auto-discovers folders. convert.py is a PARTIAL builder — do NOT use it for the real build. Using convert.py will produce a stale/incomplete site (missing section indexes and newly-discovered folders).

Deploy model: atomic swap of dist → ~/knowledgebase/site. Rollback: site.astro-prev (previous Astro build); deeper fallback site.mkdocs-bak (untouched original mkdocs). This supersedes the earlier cutover/rollback plan in 3b. Astro + Starlight rebuild — STAGED (not cut over) — the Astro site is now the live one.

6b. PARKED — new-page SEARCH indexing (~90 of 253 embedded)

New pages RENDER but are NOT yet searchable

The 253 new pages browse/render fine, but only ~90 / 253 were embedded into Qdrant manuals before the box died. hybrid.py builds its BM25 index from the Qdrant collection, so a new page becomes searchable (keyword and dense) only after its embedding completes. Until the embed finishes, the new /tsb/, /power-elect/, /general/ pages won’t show up in search.

For Agents — RESUME the embed (only once the box is stable / UPS in)

Re-run the CPU-capped embed until 0 remaining, then restart the service, then verify. index_new.py is resume-safe: deterministic IDs, skips already-embedded pages — relaunch after each reboot until done.

  1. Launch the capped embed (resume-safe): 
    sudo systemctl reset-failed kbembed
sudo systemd-run --unit=kbembed --collect \
  -p CPUQuota=400% \
  -p StandardOutput=append:/home/levander/kb-vectors/index_new.out \
  -p StandardError=append:/home/levander/kb-vectors/index_new.out \
  /home/levander/kb-vectors/embed_launch.sh
  2. When 0 remaining: sudo systemctl restart knowledgebase.service
  3. Verify search: /api/search?q=<distinctive new-page term> returns a /tsb/ / /power-elect/ / /general/ page.
  4. Run the deferred Playwright sidebar click-through proof (the sidebar-focus verification skipped in 6a).
  • Images for the new pages were NOT downloaded (binaries) — a light follow-up. The convert/build guard drops the missing refs so the build still works without them.

6c. POWER — CONFIRMED: heavy CPU triggers the brownouts

Heavy CPU load TRIGGERS the brownout/reboot on this marginal mains (upgrades §1)

This upgrades the 1. Mains power — root cause CONFIRMED, and it is damaging the SSD finding from “confirmed mains loss” to a confirmed cause→effect: running a full 24-core embedding reboots the box (brownout). CPU-capping to ≤500% / ≤5 cores stayed stable. Until the UPS is installed, keep all heavy CPU/GPU work capped (this is why the embed in §6b runs at CPUQuota=400%).

Overnight 2026-08-06 outage timeline (from journalctl --list-boots): stable 2026-08-05 21:36 → 2026-08-06 03:21 (first cut) → ~12 hard reboots 03:21–04:56 → dead 04:56–09:00 (~4 h) → back 09:00. NVMe unsafe_shutdowns still climbing (was 241 — damage is ongoing). The UPS (due Fri 2026-08-07) + NUT auto-shutdown remains the fix.

6d. PROCESS gotcha — don’t trust an empty TaskList to mean a background agent finished

An empty TaskList does NOT mean a long background agent is done

A crawl agent ran ~83 min across reboots while the TaskList showed no tasks — which led to launching overlapping agents (3-way concurrent KB writes). It converged (dedup confirmed the one new G16A page is unique, no dupes), but avoid overlapping content/deploy agents: rely on completion notifications, and never run two site-rebuild/deploy agents at once.

Graph View

Backlinks