Levandor

Agent Guide — Deprovision the VM

5 min read

DESTRUCTIVE — read the whole guide before running anything

terraform destroy removes the VM, its disk, the VPC, the firewall rule, the service account, and the IAM bindings. Bind-mount data on /var/lib/* is gone when the VM is destroyed. Re-provision creates a fresh node with a new tailnet identity. Plan accordingly.

When you’d actually do this

  • Permanent decommission (“we’re done with polymarket-infra”).
  • Hard reset (everything broken, fastest path to a clean state).
  • Region migration (already done once — see gcp-terraform-ansible-gotchas for the GCP soft-delete + Tailscale name collision traps).

For “just remove an app” → agent-guide-remove-app. For “bump an image” → agent-guide-redeploy-app. Don’t destroy the VM unless you actually mean it.

What gets destroyed (terraform-tracked)

Per terraform state list at the typical state:

google_compute_network.this                                       # the VPC
google_compute_subnetwork.this                                    # the subnet
google_compute_firewall.tailscale_direct[0]                       # the only ingress rule
google_compute_instance.this                                      # the VM + its boot disk
google_service_account.vm                                         # ops-vm-sa@<proj>...
google_project_iam_member.vm_artifact_registry_reader             # SA binding
local_file.inventory                                              # ansible inventory
google_project_service.this["compute.googleapis.com"]             # the 5 enabled APIs
google_project_service.this["artifactregistry.googleapis.com"]
google_project_service.this["iam.googleapis.com"]
google_project_service.this["iamcredentials.googleapis.com"]
google_project_service.this["sts.googleapis.com"]
module.ar_wif.google_artifact_registry_repository.this            # the apps AR repo
module.ar_wif.google_iam_workload_identity_pool.github            # WIF pool
module.ar_wif.google_iam_workload_identity_pool_provider.github_oidc  # OIDC provider
module.ar_wif.google_service_account.ci_pusher                    # CI pusher SA
module.ar_wif.google_artifact_registry_repository_iam_member.ci_pusher_writer
module.ar_wif.google_service_account_iam_member.wif_repo_binding[<each repo>]

That’s ~17 resources.

What does NOT get destroyed

  • Artifact Registry images — wait, no, the apps repo IS destroyed, which destroys all images inside it (polymarket-fetch:*, pm-arb:*, every SHA tag). Account for this if re-provision is delayed.
  • GitHub Actions secrets (TS_OAUTH_CLIENT_ID, TS_OAUTH_SECRET, POLYMARKET_FETCH_ENV) — they’re in the GH repo, not GCP.
  • Tailscale OAuth client — Tailscale-side, not in terraform state.
  • Tailscale registered nodes — terraform destroy does NOT unregister the VM’s tailnet node (gcp-terraform-ansible-gotchas #15). After destroy, the node sits as offline in the admin console. You must delete it manually OR accept the next provision will append a -1 suffix.
  • Supabase schema and data — Supabase is an external SaaS, untouched.
  • secrets/*.env files on operator laptop — local-only, untouched.
  • terraform.tfvars — local, untouched.
  • The Obsidian vault — obviously.

Soft-delete gotchas

GCP applies a 30-day soft-delete retention to some resources:

  • WIF pool (google_iam_workload_identity_pool) — name reserved for 30 days after destroy. Re-creating with the same name returns 409: Requested entity already exists. Mitigation: curl ... :undelete to revive before re-apply, then terraform import to put the resurrected resource back into state. See gcp-terraform-ansible-gotchas #23 (added 2026-05-27 EU migration).
  • OIDC provider (google_iam_workload_identity_pool_provider) — same 30-day retention, same :undelete API path.
  • Service accounts — also soft-deleted, similar 30-day retention. Less of a problem because we can also rename SAs (current pattern uses ${var.vm_name}-sa so a vm_name change avoids the collision).

If you ARE within the 30-day window of a previous destroy, expect to fight these gotchas during re-apply.

Pre-flight before destroy

# 1. Save anything you care about from the VM
ssh ops@polymarket-infra '
  sudo du -sh /var/lib/wallet-monitor /var/lib/pm-arb-paper 2>/dev/null
  sudo ls -la /var/lib/wallet-monitor /var/lib/pm-arb-paper 2>/dev/null
'
 
# 2. Back up the wallet-monitor JSONLs (recoverable — they're just append logs)
mkdir -p ~/backup/wallet-monitor-$(date +%Y%m%d)
scp 'ops@polymarket-infra:/var/lib/wallet-monitor/*.jsonl' ~/backup/wallet-monitor-$(date +%Y%m%d)/
 
# 3. Back up pm-arb paper trades JSONLs (also recoverable — pm_paper_trades is the canonical record in Supabase)
mkdir -p ~/backup/pm-arb-paper-$(date +%Y%m%d)
scp 'ops@polymarket-infra:/var/lib/pm-arb-paper/*.jsonl' ~/backup/pm-arb-paper-$(date +%Y%m%d)/
 
# 4. (Optional) revoke the Tailscale auth key BEFORE destroy
#    The current key is reusable+ephemeral so a fresh provision works with the same key.
#    Only revoke if you suspect compromise or want a clean rotation.
 
# 5. Confirm gcloud ADC + Tailscale daemon are healthy locally
make preflight

The destroy

From the terraform repo root:

cd /Users/levander/levandor/terraform
make deprovision

(Or the auto-approve form: terraform destroy -auto-approve. The Makefile target wraps terraform destroy and prompts interactively.)

Expect ~3 min. Order: VM first (slowest, ~50-60s for graceful shutdown), then SA + IAM binding (~5s each), subnet + network (~15-25s each), APIs (~1s each).

Successful end: Destroy complete! Resources: 17 destroyed.

Post-destroy cleanup

  1. Tailscale admin — delete the offline node. admin → Machines, search for polymarket-infra (or ops-vm / whatever hostname), click Delete. If you skip this, the next provision will tag the new node as polymarket-infra-1 and you’ll spend 15 min debugging “why isn’t my hostname resolving” (see gotcha #15).

  2. AR images gone — if re-provisioning, the first deploy will fail with not found until the app repo’s CI rebuilds + pushes images. Either wait for the next commit to main on each app repo, OR manually trigger gh workflow run release.yml -R wowjeeez/polymarket-fetch --ref main and gh workflow run release.yml -R <other app repo>... BEFORE running make deploy.

  3. GitHub Actions secrets stay — no action needed unless you want to rotate.

  4. Operator laptop — ~/.ssh/known_hosts entries for the old node’s Tailscale IP are stale. ssh-keygen -R polymarket-infra if ssh complains about key mismatch on next provision.

Re-provision after destroy

See agent-guide-provision-new-vm for the create flow. Quick version:

cd /Users/levander/levandor/terraform
make provision         # apply + ansible configure
make verify            # smoke check

Then redeploy apps via the CD workflow (agent-guide-redeploy-app Path 1) — but those will fail until each app repo’s CI has pushed a fresh image to the brand-new AR repo. Trigger builds first: gh workflow run release.yml -R wowjeeez/polymarket-fetch.

Don’t-do shortcuts

  • ❌ gcloud compute instances delete polymarket-infra --zone=... — out-of-band destroy. Terraform state still thinks the VM exists; next terraform apply may try to create a second one or trip over the missing instance.
  • ❌ rm -rf .terraform/ terraform.tfstate* — local state nuke. Resources persist in GCP, you lose visibility into them, future applies create duplicates. To start clean you’d then have to gcloud everything by hand.
  • ❌ Editing terraform.tfstate by hand — never. State is JSON but the schema is internal.
  • ❌ Destroying just the VM keeping the VPC — terraform doesn’t support partial destroys cleanly; just run the full destroy.

If you really need to remove just one resource: terraform state rm <addr> (drops from state without destroying in GCP), then either terraform import to re-add or leave it for cleanup. Rarely the right answer.

Graph View

Backlinks