Levandor

telep-mainframe root LUKS — TPM2 auto-unlock (clevis), passphrase kept as fallback

7 min read

Follow-up to 2026-07-23-mains-power-shutdowns: telep-mainframe’s root filesystem is LUKS2 on LVM and was passphrase-only, so after any reboot — including the ones a mains event forces — the box sat at a passphrase prompt and never came back on its own. That prompt is unreachable over SSH (there is no dropbear-initramfs), so a box that reboots on power events needs the disk to unlock unattended. Goal here: TPM-sealed auto-unlock at boot, keeping the passphrase as a fallback. Done 2026-07-24 with clevis, and PROVEN IN PRODUCTION the same day: the box self-recovered the encrypted root through a 3-reboot storm with no passphrase (see 2026-07-24-psu-load-resets).

For Agents

Disk: root LUKS2 container on /dev/nvme0n1p3 (mapper nvme1n1p3_crypt, UUID a3a8e37d-79fa-484b-bc3f-40c56df95337) → LVM → root fs. (NVMe device nodes on this box flip across reboots — always lsblk -f; see the telep-mainframe drive gotcha.) TPM: TPM2 present, /dev/tpmrm0 (MSFT0101). Secure Boot is OFF. Mechanism (working): clevis TPM2 binding. Keyslots after the work: 0 = passphrase (fallback, untouched), 1 = TPM2 (tpm2 '{"hash":"sha256","key":"ecc"}'). Check it: clevis luks list -d /dev/nvme0n1p3 (shows the tpm2 binding on slot 1), sudo cryptsetup luksDump /dev/nvme0n1p3 (both keyslots + the clevis token). Header backup taken first: ~/telep/luks-header-nvme0n1p3-20260723.img (600) on the Mac. Reboot test: ✅ PASSED in production 2026-07-24 — the box auto-unlocked the root through 3 unattended reboots (a PSU-load reboot storm, 2026-07-24-psu-load-resets). Not a lab test any more. Caveat: auto-unlock only helps once the box boots — BIOS Restore on AC Power Loss is still stay-off, so a mains cut still needs a button press.

Why not systemd-cryptenroll

The obvious modern approach — systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs= <dev> — enrolls a TPM keyslot fine, but Debian 13’s stock initramfs does not consume it. Debian uses initramfs-tools with the classic cryptsetup hook, not systemd in the initramfs; it has no idea what a systemd-tpm2 LUKS token is. update-initramfs -u prints

cryptsetup: WARNING: nvme1n1p3_crypt: ignoring unknown option 'tpm2-device'

and the box still prompts for the passphrase at boot. That approach was abandoned and the systemd-tpm2 keyslot wiped (systemd-cryptenroll --wipe-slot=tpm2 <dev>, which needs no passphrase).

Debian stock initramfs (initramfs-tools) does NOT support systemd-cryptenroll TPM2 tokens

systemd-cryptenroll --tpm2-device=… enrolls a valid TPM keyslot, but on Debian’s initramfs-tools initramfs update-initramfs warns ignoring unknown option 'tpm2-device' and the box still prompts at boot — the systemd-tpm2 token is never read. For TPM-based LUKS auto-unlock on Debian, use clevis (clevis-initramfs + clevis-tpm2), not systemd-cryptenroll + a crypttab tpm2-device= option. (The systemd path is the right one on a systemd-in-initramfs distro like Arch/Fedora; it is the wrong tool on Debian’s classic hook.)

What works: clevis

Installed clevis clevis-luks clevis-initramfs clevis-tpm2, then bound the TPM:

sudo clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'
sudo update-initramfs -u
  • Empty config '{}' = no PCR binding — deliberate. No pcr_ids means the unseal is not gated on measured-boot state, so a firmware or kernel update won’t silently break auto-unlock. With Secure Boot off the measured-boot chain isn’t trustworthy for gating anyway, so PCR binding would add fragility without meaningful assurance. (See the security tradeoff below.)
  • Result: a clevis token lands on the LUKS header and keyslot 1 becomes the TPM slot (1: tpm2 '{"hash":"sha256","key":"ecc"}'); keyslot 0 (the passphrase) is left intact as the fallback.
  • update-initramfs -u installs the clevis unlock hooks (scripts/local-top/clevis) into the initramfs, so clevis-luks-unlock auto-unseals the volume key from the TPM at boot — no prompt.

The TPM unseal path itself was independently proven earlier on this hardware via a scratch loopback LUKS test: enroll a TPM binding with empty PCRs, then systemd-cryptsetup attach unlocked the loop device with no passphrase. So the hardware mechanism (TPM seal → unseal) is known-good on this box; only the reboot-of-the-real-root test remains.

Hard-won gotchas

An orphaned systemd-crypt* process survives holding the LUKS header lock AND /dev/tpmrm0

A systemd-cryptsetup attach / systemd-cryptenroll that gets stuck at a passphrase prompt (e.g. because the caller’s SSH session was killed) does not die — it survives as an orphan that holds the LUKS header lock and /dev/tpmrm0, and it keeps re-broadcasting a systemd-ask-password request. That silently blocks any later clevis luks bind (header locked) and confuses the controlling tty. Fix: SIGKILL the orphaned systemd-crypt* PIDs (SIGTERM may not clear one mid-TPM-op), then confirm fuser /dev/nvme0n1p3 and lsof /dev/tpmrm0 are clear and cryptsetup luksDump still shows the passphrase slot, before retrying the bind.

  • Adding any new keyslot needs the existing passphrase — it cannot be done for the user. Both clevis luks bind and systemd-cryptenroll authorize the new slot against an existing keyslot, so the current passphrase must be typed by someone who knows it; an agent cannot supply it. And the operation is genuinely slow (argon2 KDF keyslot-add + TPM seal = tens of seconds) — an impatient Ctrl-C leaves a half-run process (see the orphan gotcha above). Let it finish.
  • Back up the LUKS header before touching keyslots. Done first here: cryptsetup luksHeaderBackup /dev/nvme0n1p3 --header-backup-file … → ~/telep/luks-header-nvme0n1p3-20260723.img (600, on the Mac). A corrupt header = an unrecoverable disk; the backup is cheap insurance.
  • Reversible. Remove the clevis binding with sudo clevis luks unbind -d /dev/nvme0n1p3 -s 1; a systemd TPM slot (if one were ever used) wipes with sudo systemd-cryptenroll --wipe-slot=tpm2 <dev> (no passphrase needed for the wipe). The passphrase slot 0 is never touched by any of this.

Validated in production (2026-07-24 reboot storm)

The clevis TPM2 auto-unlock was proven in a real reboot storm on 2026-07-24: the PSU browned out under load and hard-reset telep-mainframe ~3× in 30 minutes (see 2026-07-24-psu-load-resets). The box self-recovered the encrypted root through all 3 unattended reboots — systemd-cryptsetup@nvme1n1p3_crypt started on boot, root mounted, no passphrase. Every service came back on its own: the knowledgebase, kb-qdrant (2026-07-24-kb-vectorize-complete), the home + knowledgebase tailscale nodes (2026-07-24-global-dashboard), the Frigate container, camwall, wifi-usage, and the intruder-alarm. Reboot-test status therefore changes from “pending” to “PASSED in production”.

Incomplete / complementary (honest state)

  • Auto-unlock only helps once the box actually BOOTS. telep-mainframe’s BIOS Restore on AC Power Loss is still set to stay off, so after a mains cut the box stays powered off until someone presses the button — the clevis unlock never even runs. Full unattended power recovery still needs the BIOS flip to Power On (physical access) and ideally a UPS (the real fix). See 2026-07-23-mains-power-shutdowns.
  • Security posture — the accepted tradeoff. With empty PCRs + Secure Boot off, the disk auto-unlocks on this hardware. That protects against bare-drive theft (pull the NVMe → the TPM-sealed key isn’t on it, so it won’t unlock in another machine) but not whole-box theft (a thief who powers the whole machine on gets an unlocked disk). This was the deliberate tradeoff for unattended reboot; the passphrase slot remains as the human fallback.

Security hygiene note

During the keyslot work a passphrase-echo hygiene issue occurred (the LUKS passphrase was at risk of being echoed/logged); rotation of the passphrase was suggested as a precaution. No secret values are recorded in this vault — see the rotate-these list in SESSION-HANDOVER.

Graph View

Backlinks