Levandor

Arrow Flight Dev Deploy Runbook (recurring facts) — BE-1595

3 min read

The recurring, reusable facts for deploying the Arrow Flight streaming feature to dev for the mando BESS platform — the three artifacts that must move together, the IaC location, the harmless-red CI step, the runtime switch, and where to verify. Distilled from a multi-session deploy.

For Agents — what moves together

A Flight deploy is three artifacts: (1) the mando server image, (2) the py-mando wheel, (3) the consumer images (optimization + forecast, both rebuilt on the matching wheel). When a wire type changes, all three must come from the same build — see be-1595-flight-wire-type-version-skew-2026-06-24. The consumer wheel-lock mechanics are in be-1595-arrow-consumer-lock-script-2026-06-24.

Migration drift → mando crash-loop

"V…__… is missing from the filesystem" = migration drift

mando crash-loops when the DB has migrations the branch’s source tree doesn’t (Flyway-style: a version applied to the DB has no matching file on the branch). Symptom in logs: V…__… is missing from the filesystem.

Fix: merge develop into the deploy branch so the branch’s migration set ⊇ the DB’s. This is routine on long-lived feature/*arrow* branches that drift behind develop.

IaC location and image pins

The infrastructure lives in optimization-universe-iac:

  • Worktree: .worktrees/mando-arrow
  • Terraform var file: terraform.auto.tfvars.json — holds the image pins for mando, optimization-algo, and forecast-algo.

To bump the deployed images, edit the three pins in terraform.auto.tfvars.json (in the mando-arrow worktree) and apply.

terraform_apply:dev always goes red on a customer-portal S3 403 — harmless

The terraform_apply:dev CI step always shows red because of a pre-existing customer-portal S3 403 (HeadObject) that is not ours. Terraform still prints Apply complete! — the red is cosmetic, not a failed apply. Do NOT chase it; confirm Apply complete! in the log and move on.

Runtime switch (algo task defs)

Flip the consumers onto Flight via env on the algo task defs:

VarValueDefault
MANDO_FETCH_STRATEGYflightrest
MANDO_FLIGHT_HOST(mando host — localhost when co-located)
MANDO_FLIGHT_PORT(Flight gRPC port, 50051)

See be-1595-enabling-arrow-flight-consumers-2026-06-16 for the full per-service topology (co-located vs separate task) and the silent-REST-fallback gotcha.

Verify in Datadog

Datadog EU, env:dev. Services:

  • bess-os-service-mando
  • bess-os-algo-optimization
  • bess-os-algo-forecast

What "working" looks like in the logs

  • optimization on Flight: clean, fully on the Flight path (cache-window fetches only).
  • forecast on Flight: emits falling back to REST (gRPC Unimplemented) for the historical portion, then completes — this is the expected graceful degradation, not a failure. See be-1595-flight-rest-graceful-degradation-2026-06-24.
  • Remember: a “clean” run with NO logs proves nothing — a Flight misconfig silently falls back to REST on transport errors (polars.rs:234). Confirm the Flight path is actually exercised.

Deploy checklist

For Agents — dev deploy order

  1. Merge develop into the deploy branch (avoid migration drift crash-loop).
  2. Build the three artifacts from the same build if any wire type changed (mando image + py-mando wheel + consumer images).
  3. Lock the consumers to the matching wheel (be-1595-arrow-consumer-lock-script-2026-06-24) — re-copy the one-liner fresh, verify the echoed wheel.
  4. Bump the three image pins in optimization-universe-iac/.worktrees/mando-arrow terraform.auto.tfvars.json; apply (ignore the harmless S3-403 red; confirm Apply complete!).
  5. Set MANDO_FETCH_STRATEGY=flight + MANDO_FLIGHT_HOST/PORT on the algo task defs.
  6. Verify in Datadog EU env:dev (mando / optimization / forecast). Expect forecast falling back to REST, optimization fully on Flight.

Graph View

Backlinks