How a mando build actually reaches an environment: mando CI publishes docker images, and optimization-universe-iac (terraform, ECS) decides which version runs where. Originally verified 2026-07-13; re-verified 2026-08-18 against origin/develop @ 9600b34c of optimization-universe-iac, which corrected the pin model (see the callout below). The same material is recorded as section 14 (“Deployment - getting a build onto an environment”) of the untracked mando AGENTS.md (see Mando AGENTS.md Master Guide) - AGENTS.md sections 14.1, 14.2, 14.4 and 14.7 still carry the old model and are stale.
CORRECTED 2026-08-18 - the dev_test shared-pin group is GONE
Everything that says pins live in terraform/terraform.auto.tfvars.json under components.dev_test.mando.version is wrong since IaC commit 3f50c77 (2026-07-14, Jozsef Nagy, “feat: migrate variables from cicd config and auto.tfvars to separate env.tfvars files”). That file no longer exists in terraform/. Pins now live in per-environment files: terraform/environments/{dev,test,int,prod}.tfvars, each with its own flat components = { mando = { version = "..." } ... } block.
dev and test are now independently pinnable and have diverged in practice. Bumping one does NOT move the other.
For Agents
Never deploy from mando alone. Deployment is a two-repo ceremony: mando CI only publishes images to ECR; nothing runs until optimization-universe-iac bumps the version pin in terraform/environments/<env>.tfvars AND a human manually triggers terraform_apply:{env} (still when: manual, .terraform_apply template line 213).
Two-repo ceremony
mando (.gitlab-ci.yml): builds and publishes docker images to ECR.
optimization-universe-iac (terraform, ECS): decides which image version runs in which environment.
Publishing an image changes nothing by itself; the IaC version pin has to move and the apply has to be triggered.
Environment matrix (current, verified 2026-08-18)
4 environments, 4 independent tfvars files. There is no component-group indirection any more: terraform/main.tf reads var.components["mando"].version and var.ecr_repos["mando"] FLAT, and local.environment_groupno longer exists anywhere under terraform/ (git grep environment_group origin/develop -- terraform/ returns nothing).
ALL terraform_apply jobs are when: manual (.terraform_apply sets it once).
The file is selected by GitLab’s environment name, not by a terraform expression:
terraform plan -var-file=environments/${CI_ENVIRONMENT_NAME}.tfvars -out=$LOCALDIR_ARTIFACTS/$TF_PLAN
CI_ENVIRONMENT_NAME comes from the environment: name: key of the .environment-vars:{dev|test|int|prod} anchor each job extends.
Read pins from origin/develop, never from the working tree
The local clone /Volumes/bandi/coding/poc/optimization-universe-iac sits on rc/1.11.0 from 2026-05-27 and is badly stale - it still contains the pre-migration layout. Canonical read:
git -C /Volumes/bandi/coding/poc/optimization-universe-iac fetch origin developgit -C /Volumes/bandi/coding/poc/optimization-universe-iac show origin/develop:terraform/environments/test.tfvars
int_prod naming trap (historical)
The int_prod group name is gone from the code, but the trap survives in older notes and in AGENTS.md section 14: int_prod always meant INT, not real prod. Do not treat historical int_prod values as production evidence. Cf. BE-1597 DuckDB UI Exposure Research.
Evidence that dev and test really do diverge
IaC commit 16e8241 (Gabor Nagy, 2026-08-11 14:10 CEST, “feat: bump components version”) touched bothdev.tfvars and test.tfvars, 6 lines each. Immediately before it:
1.17.0-dev.2729271578.24a0f34b (mando develop 24a0f34b, “Merge branch ‘main’ into develop”, 2026-08-04)
16e8241 set both to the same value; they were two different builds before that.
Deploy to dev, step by step
Merge to mando develop. The Publish Service Docker Dev job (its only: list is exactly develop and rc/*) pushes an image tagged {version}-dev.{pipeline_id}.{short_sha}. The + of APP_VERSION becomes - in docker tags, e.g. 1.11.0-dev.2555942905.50575825.
In optimization-universe-iac on the develop branch, edit terraform/environments/dev.tfvars: set components.mando.version to the new tag. If you also want test on that build, edit terraform/environments/test.tfvars in the same commit - they are separate pins and neither follows the other. Observed commit convention in that repo: feat: bump component versions / feat: bump components version.
terraform_plan:dev runs on push. Review the plan artifact; the expected change is the mando container image in the bess-os ECS task definition ONLY. A bigger plan is not automatically danger: task definitions are immutable, so every affected one shows as a “must be replaced” destroy+create pair, and any IaC develop drift the env has not received yet rides along (worked example: 17 add / 19 change / 5 destroy on the Dev Deploy 2026-07-14 (error telemetry), where the extras were the bess_os_frincoming_sync lambda stack + 2 RDS analytics SG rules).
Manually trigger terraform_apply:dev.
Feature-branch deploy trick
mando publishes only from develop and rc/*. To get a feature branch onto dev: temporarily add the branch name to the only: list of Publish Service Docker Dev in mando .gitlab-ci.yml, let it publish, then bump the tfvars pin as above.
The only: edit alone is NOT enough in mando itself: Publish Service Docker Dev carries a needs/dependencies entry on PyMando Win Dev, which does not exist on feature branches; the dangling need kills pipeline creation outright. Comment that entry out in the same temporary CI commit.
The IaC side has its own counterpart trick: terraform_apply:dev exists ONLY on the IaC develop branch (feature branches get plan only). To apply from an IaC feature branch, add the branch to terraform_apply:dev’s only: list in its own temporary commit (observed: “ci: enable dev apply from error-hdl branch”); strip it if the branch ever merges.
Runtime config without rebuild
CORRECTED 2026-08-18 - runtime config moved too
The old claim (“per-env mando config lives in optimization-universe-iac .gitlab-ci.yml under .environment-vars:{env}”) is stale since 3f50c77. Those anchors now carry ONLY ASSUME_ROLE_ARN, ACCOUNT_ID, ACCOUNT_NAME, AWS_REGION and environment: name:. The .terraform template says it verbatim: “Secrets and CI-dynamic values - all other variables are in environments/.tfvars”. Grepping SIMULATION_MODE, MANDO_FLOW_SCHEDULE_*, EBS_SMB_HOST or MANDO_FLOW_ACTIVE_VERSION in the IaC .gitlab-ci.yml now returns nothing.
Per-env mando runtime config lives in terraform/environments/<env>.tfvars as lower-case terraform variables:
Note the shape change: sizing and active-version are now per region inside bess_os_regions, not flat per env. dev carries both fi and fr; test carries fi only (relevant to fr-region-missing-datadog-logs-2026-07-21).
Changing any of these is still an IaC-only change: edit the tfvars file, plan, manual apply. No mando rebuild.
mando_flow_active_version constraint
bess_os_regions.<region>.mando_flow_active_version must name a setup version that exists in mando config/flows/manifest.yaml (the flow deployment catalog, see Mando AGENTS.md Master Guide). Bump the manifest and the tfvars value together. Observed 2026-08-18: dev fi = V1_4, dev fr = V1_4_SIMULATION.
Post-deploy verification
GET https://mando.{env-domain}/version must equal the bumped tag (the endpoint also returns component versions).
/health returns 200; swagger loads.
Datadog sidecar logs filtered on error.kind, plus the refinery migration lines at boot. A failed migration is the most common bad deploy (cf. the migration-drift crash-loop in be-1595-arrow-flight-dev-deploy-runbook-2026-06-24).
RESOLVED 2026-07-24 (was: standing dev hazard, bess-am rows in dev's refinery history)
The two files (U202605131400__create_event_table + U202605141900__create_kinesis_checkpoint_table) landed on origin/develop via the bess-am POC merge (3ebcc897, merge 6fae64a5, 2026-07-23), so develop-built images now carry them and no longer crash-loop on dev. The byte-exact restore step is obsolete. Details: Dev Deploy 2026-07-14 (error telemetry); repo docs/superpowers/HANDOVER-error-handling-2026-07-24.md item 4.
dev runs SIMULATION_MODE=true.
Scheduled flows should land Success/Warning, not Fatal.
Local pre-deploy E2E script: test/pi1/end-to-end.py.
Workspace version on mando origin/develop is 1.17.0. Deployment state and consequences for the error-handling chain: Test Env Deployment State 2026-08-18.
Mando AGENTS.md Master Guide - AGENTS.md section 14 records this ceremony; sections 14.1/14.2/14.4/14.7 are stale as of 2026-08-18 (dev_test group, auto.tfvars.json, .environment-vars runtime config, local.environment_group gating)