Spec 1 Multi-VM Refactor (2026-06-30)

For Agents

On 2026-06-30 the levandor-infra Terraform root was refactored from a single-VM design (module "vm") to a multi-VM design (module "vm" { for_each = var.vms }). This closed HANDOVER follow-up #2 (state cleanup from the Frankfurt → Zurich migration) and added a second VM (fk-dev) in one bundled change. Apply result: 7 added, 2 changed, 0 destroyed. Production pmv2-zurich containers verified untouched.

The refactor that “agent-guide-provision-new-vm said was needed for two concurrent VMs” — now landed. Bundled with the fk-dev provision so the resource address renames + state-mv surgery happened once, not twice.

TL;DR

Aspect	Before	After
VM instantiation	module "vm" (one) + transient module "vm_zurich" graft	module "vm" { for_each = var.vms }
Shared infra (VPC, SA, egress IP)	Resources at root level, baked ${var.vm_name} into names	Self-contained inside each module.vm[<key>] iteration
Inventory template	Single [vm] host group	Per-VM entry + parent :children groups
Ansible site.yml	Single play on hosts: vm running all 8 roles	Per-host-group plays (pmv2_full, fk_dev_base, …)
Outputs	Scalars (ssh_command, vm_external_ip)	Maps keyed by VM name
State cleanliness	terraform plan reported “3 to add” phantoms	Clean plan; state reflects reality

Why this pattern won

The historian flagged that HANDOVER’s pending follow-up #2 (state cleanup after the Frankfurt → Zurich migration) was a superset of what fk-dev’s provisioning needed — same edits to the same resource address names, same state-mv surgery. Bundling them avoided two sequential PRs touching the same surface twice and let one round of state mv operations cover both intents.

Bundle-when-the-surface-overlaps heuristic

If an open cleanup task and a new feature both rewrite the same resource addresses, do them in one operation. The state-mv cost is fixed per address, not per intent.

Before the refactor

# main.tf (excerpt — pre-refactor)
module "vm" {
  source = "./modules/vm"
  vm_name = var.vm_name
  # …
}
 
module "vm_zurich" {
  source = "./modules/vm"
  # transient graft from the Frankfurt → Zurich migration
  # …
}
 
resource "google_compute_subnetwork" "zurich" {
  # orphan block — referenced by vm_zurich, never moved into the module
}
 
resource "google_compute_network" "this" {
  name = "${var.vm_name}-net"
  # baked vm_name into shared infra → collision risk on second VM
}
 
resource "google_service_account" "vm" {
  account_id = "${var.vm_name}-sa"
  # ditto
}

State problem: the original blue VM (Frankfurt) had been terraform state rm’d during the Zurich migration but main.tf still referenced its config. terraform plan reported “3 to add” phantoms for resources whose state was gone but whose HCL was still alive.

inventory.tftpl:

[vm]
${vm_name} ansible_host=${tailscale_hostname} ansible_user=ops

Single host group. ansible/site.yml:

- hosts: vm
  gather_facts: false
  pre_tasks: [...]
  roles:
    - base
    - docker
    - github_keys
    - monitoring
    - apps
    - babylon
    - nats
    - bsc_exploit

All 8 roles on every host. Multi-VM by design was aspirational, never landed.

After the refactor

Variable shape

# variables.tf
variable "vms" {
  type = map(object({
    machine_type     = string
    region           = string
    zone             = string
    image            = string
    disk_size        = number
    disk_type        = string
    subnet_cidr      = string
    tailscale_authkey = string
    tailscale_tag    = string
    startup_template = string
    ansible_groups   = list(string)
    network_name     = optional(string)
    subnet_name      = optional(string)
    sa_account_id    = optional(string)
  }))
}

Optional name-overrides (network_name, subnet_name, sa_account_id) exist for one reason: pmv2-zurich’s VPC, subnet, and SA were renamed-but-not-destroyed across the Frankfurt → Zurich migration. Forcing destroy+recreate to align the new for_each-derived names would have nuked the production VPC. The overrides let pmv2-zurich keep its legacy names while fk-dev gets the new for_each-derived defaults (fk-dev-net, fk-dev-subnet, fk-dev-sa).

Root module

# main.tf (post-refactor, conceptual)
module "vm" {
  source   = "./modules/vm"
  for_each = var.vms
 
  vm_name           = each.key
  machine_type      = each.value.machine_type
  region            = each.value.region
  zone              = each.value.zone
  # … all per-VM params from each.value …
 
  # optional name-overrides
  network_name = each.value.network_name
  subnet_name  = each.value.subnet_name
  sa_account_id = each.value.sa_account_id
}
 
resource "google_storage_bucket_iam_member" "babylon_backup_vm_writer" {
  for_each = {
    for k, v in var.vms : k => v
    if contains(v.ansible_groups, "pmv2_full")
  }
  bucket = google_storage_bucket.babylon_backup.name
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${module.vm[each.key].service_account_email}"
}

The babylon_backup_vm_writer filter is the sensible-scoping reference point: only VMs whose ansible_groups contains pmv2_full get the GCS write grant. fk-dev does NOT get babylon backup write access.

VM module

modules/vm/ now self-contains:

• google_compute_network
• google_compute_subnetwork
• google_service_account + IAM binding
• google_compute_address (egress)
• google_compute_firewall (Tailscale direct)
• google_compute_instance

Each for_each iteration produces a fully isolated per-VM stack. No shared infra at the root level.

Inventory template

{% for name, vm in vms.items() %}
[{{ name }}]
{{ name }} ansible_host={{ vm.tailscale_hostname }} ansible_user=ops
 
{% for group in vm.ansible_groups %}
[{{ group }}:children]
{{ name }}
{% endfor %}
{% endfor %}

Each VM becomes its own host group (by name) AND a member of the :children parent group(s) named in its ansible_groups.

Ansible site.yml

- hosts: pmv2_full
  gather_facts: false
  pre_tasks: [...]
  roles: [base, docker, github_keys, monitoring, apps, babylon, nats, bsc_exploit]
 
- hosts: fk_dev_base
  gather_facts: false
  pre_tasks: [...]
  roles: [base, docker, github_keys, monitoring]

Per-host-group plays. New host groups can be added without touching existing plays — the inventory template surfaces them automatically from ansible_groups.

Outputs

output "ssh_commands" {
  value = { for k, m in module.vm : k => "ssh ops@${m.tailscale_hostname}" }
}
 
output "vm_external_ips" {
  value = { for k, m in module.vm : k => m.external_ip }
}
 
output "vm_names" {
  value = { for k, m in module.vm : k => m.vm_name }
}
 
output "vm_tailscale_hostnames" {
  value = { for k, m in module.vm : k => m.tailscale_hostname }
}

All outputs migrated from scalars to maps keyed by VM name. Operator commands terraform output -raw ssh_commands now requires picking a key.

Makefile

VM ?= pmv2-zurich
 
ssh:
	ssh ops@$$(terraform output -json vm_tailscale_hostnames | jq -r '."$(VM)"')
 
verify:
	# … against $(VM) …

make ssh defaults to pmv2-zurich for backwards compatibility. make ssh VM=fk-dev for the new VM.

State surgery

The Frankfurt → Zurich migration had left state in a half-renamed condition. The full set of terraform state mv operations executed during the refactor:

terraform state mv 'google_compute_network.this'                                  'module.vm["pmv2-zurich"].google_compute_network.this'
terraform state mv 'google_compute_subnetwork.zurich'                             'module.vm["pmv2-zurich"].google_compute_subnetwork.this'
terraform state mv 'google_service_account.vm'                                    'module.vm["pmv2-zurich"].google_service_account.vm'
terraform state mv 'google_project_iam_member.vm_artifact_registry_reader'        'module.vm["pmv2-zurich"].google_project_iam_member.vm_artifact_registry_reader'
terraform state mv 'module.vm_zurich.google_compute_instance.this'                'module.vm["pmv2-zurich"].google_compute_instance.this'
terraform state mv 'module.vm_zurich.google_compute_firewall.tailscale_direct[0]' 'module.vm["pmv2-zurich"].google_compute_firewall.tailscale_direct[0]'
terraform state mv 'google_storage_bucket_iam_member.babylon_backup_vm_writer'    'google_storage_bucket_iam_member.babylon_backup_vm_writer["pmv2-zurich"]'
 
terraform state rm  'google_compute_subnetwork.this' 'google_compute_address.egress[0]'

The two state rms purged residual references left over from the Frankfurt → Zurich migration that no longer corresponded to any HCL block.

Always back up state before mv

The repo carries tfstate-backup-1782587258.json — a snapshot taken before this surgery. State mv is destructive to the state file. The provider has no rollback for a botched mv sequence other than restoring the snapshot.

Apply result

After state surgery + new HCL in place:

Plan: 7 to add, 2 to change, 0 to destroy.

• 7 added — fk-dev’s VPC, subnet, SA, IAM binding, egress address, firewall rule, compute instance.
• 2 changed — minor drift fixes on pmv2-zurich’s resources (now living inside their new module address but otherwise identical configs).
• 0 destroyed — pmv2-zurich’s 14 production containers verified untouched after apply (8 apps + 3 babylon + 3 nats sidecars).

Production safety verification

Post-apply checks on pmv2-zurich:

make ssh VM=pmv2-zurich
sudo systemctl list-units 'apps-*.service' --state=active | wc -l   # 8 app services up
sudo systemctl is-active babylon-compose.service                     # active
sudo systemctl is-active nats.service                                # active
docker ps --format '{{.Names}}'                                      # 14 containers

All green. The terraform state mv operations preserved the underlying GCP resources — only the state-file addresses changed.

Why this pattern works for future VMs

Adding a third VM is now a config-only operation:

# terraform.tfvars
vms = {
  "pmv2-zurich" = {
    # … existing config …
    ansible_groups = ["pmv2_full"]
    network_name   = "ops-vm-net"      # legacy override
    subnet_name    = "ops-vm-subnet"   # legacy override
    sa_account_id  = "ops-vm-sa"       # legacy override
  }
  "fk-dev" = {
    # … fk-dev config …
    ansible_groups = ["fk_dev_base"]
    # no name overrides — uses for_each-derived defaults
  }
  "new-vm" = {
    # … new VM config …
    ansible_groups = ["some_new_group"]
  }
}

Then ansible/site.yml gets a new play for hosts: some_new_group, and make provision does the rest.

When provisioning a brand-new VM

Read agent-guide-provision-new-vm (now updated to reflect this for_each pattern). For brand-new functionality on a brand-new VM, also create a new entry in ansible_groups so its play in site.yml stays independent of existing plays.

Decision: keep pmv2-zurich’s legacy resource names

The optional name-overrides (network_name, subnet_name, sa_account_id) exist solely to avoid destroying pmv2-zurich’s production VPC. Forcing all VMs to use for_each-derived names would have meant terraform destroy + apply for pmv2-zurich’s VPC/SA — production-affecting churn for no real gain.

The overrides are a migration accommodation, not a permanent feature. A future operator may decide to rename the production VPC and remove the overrides — the state-mv pattern is now well-understood. For now: don’t touch what’s working.

Closes HANDOVER follow-up #2

The HANDOVER document tracked five outstanding items after the Frankfurt → Zurich migration. Follow-up #2 was “clean up state and resource address inconsistencies from the migration”. The multi-VM refactor closes #2 because:

• All resources are now correctly addressed.
• The orphan google_compute_subnetwork.zurich is gone.
• The transient module.vm_zurich graft is gone.
• terraform plan no longer reports phantoms.
• State drift between HCL and reality has been resolved.

See spec-2-roadmap for the updated tracking table.

Related

• fk-dev-vm — the new VM that bundled with this refactor
• agent-guide-provision-new-vm — operator how-to (updated to reflect for_each pattern)
• levandor-infra — project overview
• gcp-terraform-ansible-gotchas — gotcha #39 (tailscaled-restart handler drops SSH session) surfaced during fk-dev provisioning
• gcp-vm-provisioning-design — original Spec 1 design
• spec-2-roadmap — roadmap status updated
• spec-1-retrospective — original retrospective
• LOG
• TOPICS