Levandor

Mando Deployment Ceremony

5 min read

Mando Deployment Ceremony

How a mando build actually reaches an environment: mando CI publishes docker images, and optimization-universe-iac (terraform, ECS) decides which version runs where. All facts verified 2026-07-13 from code in /Volumes/bandi/coding/poc/optimization-universe-iac and mando’s .gitlab-ci.yml; the same material is recorded as section 14 (“Deployment - getting a build onto an environment”) of the untracked mando AGENTS.md (see Mando AGENTS.md Master Guide).

For Agents

Never deploy from mando alone. Deployment is a two-repo ceremony: mando CI only publishes images to ECR; nothing runs until optimization-universe-iac bumps the version pin in terraform/terraform.auto.tfvars.json AND a human manually triggers terraform_apply:{env}.

Two-repo ceremony

  • mando (.gitlab-ci.yml): builds and publishes docker images to ECR.
  • optimization-universe-iac (terraform, ECS): decides which image version runs in which environment.
  • Publishing an image changes nothing by itself; the IaC version pin has to move and the apply has to be triggered.

Environment matrix

4 environments map to 2 component groups in terraform/main.tf:

EnvironmentComponent groupIaC branch rulesECR
dev, testdev_testplan + apply from develop (plan also runs on feature/* and bugfix/* branches)843164609896.dkr.ecr.eu-central-1.amazonaws.com/poc/mando/deploy
intint_prodrelease/*748634852998.dkr.ecr.eu-central-1.amazonaws.com/poc/mando
prodint_prodmain748634852998.dkr.ecr.eu-central-1.amazonaws.com/poc/mando

ALL terraform_apply jobs are when: manual.

int_prod naming trap

Reconfirmed: int_prod is historically INT, not real prod. The name is misleading; do not treat the int_prod group as production evidence. Matches the main.tf int+prod mapping already noted in BE-1597 DuckDB UI Exposure Research.

Deploy to dev, step by step

  1. Merge to mando develop. The Publish Service Docker Dev job (its only: list is exactly develop and rc/*) pushes an image tagged {version}-dev.{pipeline_id}.{short_sha}. The + of APP_VERSION becomes - in docker tags, e.g. 1.11.0-dev.2555942905.50575825.
  2. In optimization-universe-iac on the develop branch, edit terraform/terraform.auto.tfvars.json: set components.dev_test.mando.version to the new tag. Observed commit convention in that repo: feat: bump component versions.
  3. terraform_plan:dev runs on push. Review the plan artifact; the expected change is the mando container image in the bess-os ECS task definition ONLY. A bigger plan is not automatically danger: task definitions are immutable, so every affected one shows as a “must be replaced” destroy+create pair, and any IaC develop drift the env has not received yet rides along (worked example: 17 add / 19 change / 5 destroy on the Dev Deploy 2026-07-14 (error telemetry), where the extras were the bess_os_fr incoming_sync lambda stack + 2 RDS analytics SG rules).
  4. Manually trigger terraform_apply:dev.

Feature-branch deploy trick

mando publishes only from develop and rc/*. To get a feature branch onto dev: temporarily add the branch name to the only: list of Publish Service Docker Dev in mando .gitlab-ci.yml, let it publish, then bump the tfvars pin as above.

Revert the CI commit before the MR merges

The only: edit MUST be dropped/reverted before the MR merges. Reviewers reject unrelated changes (see Mando MR Review Patterns), and leaving it in would publish an image on every future push. The Python consumer repos need the same kind of override, plus an optional: true Test dependency: be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16, scripted in be-1595-arrow-consumer-lock-script-2026-06-24.

Two additions field-verified 2026-07-14 (worked example: Dev Deploy 2026-07-14 (error telemetry)):

  • The only: edit alone is NOT enough in mando itself: Publish Service Docker Dev carries a needs/dependencies entry on PyMando Win Dev, which does not exist on feature branches; the dangling need kills pipeline creation outright. Comment that entry out in the same temporary CI commit.
  • The IaC side has its own counterpart trick: terraform_apply:dev exists ONLY on the IaC develop branch (feature branches get plan only). To apply from an IaC feature branch, add the branch to terraform_apply:dev’s only: list in its own temporary commit (observed: “ci: enable dev apply from error-hdl branch”); strip it if the branch ever merges.

Runtime config without rebuild

Per-env mando config lives in optimization-universe-iac .gitlab-ci.yml, in .environment-vars:{env} blocks that flow to TF_VAR_*:

  • MANDO_FLOW_SCHEDULE_* crons
  • MANDO_FLOW_ACTIVE_VERSION
  • EBS_SMB_HOST
  • SIMULATION_MODE
  • CPU/memory
  • notification channels

Changing any of these is an IaC-only change: edit the block, plan, manual apply. No mando rebuild.

MANDO_FLOW_ACTIVE_VERSION constraint

MANDO_FLOW_ACTIVE_VERSION (currently V1_4 in all envs) must name a setup version that exists in mando config/flows/manifest.yaml (the flow deployment catalog, see Mando AGENTS.md Master Guide). Bump the manifest and the env var together.

Post-deploy verification

  • GET https://mando.{env-domain}/version must equal the bumped tag (the endpoint also returns component versions).
  • /health returns 200; swagger loads.
  • Datadog sidecar logs filtered on error.kind, plus the refinery migration lines at boot. A failed migration is the most common bad deploy (cf. the migration-drift crash-loop in be-1595-arrow-flight-dev-deploy-runbook-2026-06-24).

RESOLVED 2026-07-24 (was: standing dev hazard, bess-am rows in dev's refinery history)

The two files (U202605131400__create_event_table + U202605141900__create_kinesis_checkpoint_table) landed on origin/develop via the bess-am POC merge (3ebcc897, merge 6fae64a5, 2026-07-23), so develop-built images now carry them and no longer crash-loop on dev. The byte-exact restore step is obsolete. Details: Dev Deploy 2026-07-14 (error telemetry); repo docs/superpowers/HANDOVER-error-handling-2026-07-24.md item 4.

  • dev runs SIMULATION_MODE=true.
  • Scheduled flows should land Success/Warning, not Fatal.
  • Local pre-deploy E2E script: test/pi1/end-to-end.py.

Observed state (2026-07-13)

Current IaC checkout observed on branch rc/1.11.0:

Component groupmando version pin
dev_test1.11.0-dev.2555942905.50575825
int_prod1.10.0-2533146896.b06b6b59

Update 2026-07-14: dev_test bumped to 1.16.1-feat.2674882859.6bc418cd via the temporary IaC branch feature/error-hdl (BE-3482 telemetry deploy, Dev Deploy 2026-07-14 (error telemetry)). Before that bump dev was nominally on 1.15.0-dev.2655042609.3024e20c, with live task labels showing 1.15.0-feat.2664534358.627578d7 from an earlier hand-applied feature deploy.

Graph View

Backlinks