Dev Deploy 2026-07-14 (error telemetry)

Hand-applied dev deploy of the BE-3482 error/log telemetry (the MR !569 content) via a throwaway mando branch plus a temporary IaC branch, while !569 awaits review. The first apply crash-looped mando on refinery migration drift caused by unmerged BE-2262 bess-am migrations already recorded in dev’s database; restoring the two migration files byte-exact fixed it. The crash itself became the first live validation of the new error telemetry.

RESOLVED 2026-07-24: was a standing crash-loop hazard for develop-based dev images

The two migrations landed on origin/develop via 3ebcc897 “feat: add bess-am POC” (feature/BE-2262-bess-am-poc, merge 6fae64a5, 2026-07-23). develop’s mando-bess/database/ file set is now byte-identical to the throwaway dev-deploy commit 6bc418cd (git ls-tree diff, verified 2026-07-24), so any develop-built image now carries a superset of what dev’s refinery_schema_history has applied and no longer crash-loops. No action was needed and the byte-exact restore workaround below is obsolete. Repo cross-ref: docs/superpowers/HANDOVER-error-handling-2026-07-24.md item 4. Was (now cleared): Dev’s refinery_schema_history contains U202605131400__create_event_table AND U202605141900__create_kinesis_checkpoint_table from the (then-unmerged) BE-2262 bess-am/kinesis POC branches (deployed to dev in May as a feat build). Refinery validated the DB’s applied migrations against the files shipped in the image, so ANY image built without those two files (i.e. any develop-based image) failed validation at boot and crash-looped.

Goal and starting state

• Goal: get the new BE-3482 error/log telemetry (fingerprint activity + version-fallback panic fix, the content of MR !569) flowing on dev while the MR awaits review.
• Dev was nominally running the pre-conformance build 1.15.0-dev.2655042609.3024e20c; the live task labels actually showed 1.15.0-feat.2664534358.627578d7 from an earlier hand-applied feature deploy.

Mechanism: throwaway mando branch

Branch feature/BE-3482-dev-deploy, never to be merged:

• Base: origin/develop tip 51d2b516.
• Cherry-picked the two !569 commits (fingerprint activity + version-fallback panic fix).
• One CI commit with two edits to .gitlab-ci.yml:
  • added the branch to the only: list of Publish Service Docker Dev (the standard trick, Feature-branch deploy trick);
  • commented out the PyMando Win Dev entry in needs/dependencies. That job does not exist on feature branches, and a dangling needs entry kills pipeline creation outright (same failure class as be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 in the consumer repos).

IaC side: temporary apply-enabled branch

In optimization-universe-iac, branch feature/error-hdl off its develop:

1. Bumped components.dev_test.mando.version in terraform/terraform.auto.tfvars.json.
2. Second commit “ci: enable dev apply from error-hdl branch”: added the branch to terraform_apply:dev’s only: list. The apply job normally exists ONLY on develop; feature branches get plan only.

Both IaC commits are temporary

The version bump gets superseded and the only: edit MUST be stripped if feature/error-hdl is ever merged.

Crash-loop: refinery migration drift (dev DB ahead of develop)

Symptoms

The first apply crash-looped mando at boot with:

MigrationError::PatchApplicationFailed: migration V202605131400__create_event_table is missing from the filesystem

Root cause

Dev’s refinery_schema_history contains two applied migrations from the unmerged BE-2262 bess-am/kinesis POC branches (feature/BE-2262-bess-am-poc etc., see BESS AM (BE-2262) - mando-bess-am), deployed to dev in May as a feat build:

• U202605131400__create_event_table
• U202605141900__create_kinesis_checkpoint_table

Any image whose filesystem lacks those files fails refinery validation at boot. The deploy branch was cut from the develop tip, and develop has never contained them.

For Agents: this is the INVERSE of the BE-1595 drift case

In be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 the deploy branch was BEHIND develop and merging develop fixed it. Here the DB is ahead of develop itself (unmerged feature migrations were applied to dev), so merging develop can NEVER fix it. The only fixes are shipping the files in the image or cleaning the DB rows.

Fix

• Restored both migration files byte-exact as commit 6bc418cd “fix: restore bess-am migrations present in the dev database”. The two files have a single canonical git blob (f45b6dd4) across ALL source commits, so a byte-exact restore satisfies refinery’s checksum validation.
• New image 1.16.1-feat.2674882859.6bc418cd; re-bumped the IaC pin; Andras triggered the apply.
• Result: Apply complete!, resources 4 added / 19 changed / destroys 5 → 4 versus the first apply. Task-definition replacements are mechanical destroy+create, and the first apply had already created the 12 genuinely new resources (see the plan-reading lesson below).

Plan-reading lesson: big plans on a version bump can be mechanical

The first apply’s plan showed 17 add / 19 change / 5 destroy for what was “just a version bump”. That was NOT danger:

• The destroys were all task definitions marked “must be replaced”: task defs are immutable, so every affected one shows as a destroy+create pair.
• 12 resources were genuinely new but unrelated: IaC develop drift catch-up the dev environment had not yet received, namely the bess_os_fr incoming_sync lambda stack and 2 RDS analytics SG rules.

The minimal-plan expectation in Mando Deployment Ceremony (mando container image only) holds only when IaC develop has NOT drifted ahead of the environment.

Error telemetry validation (observed from the crash itself)

The crash-loop logs were the first live sample of the BE-3482 telemetry on dev, and they conform to the standard:

Field	Observed	Verdict
error.code == error.kind == error.type	all three carry the same value	Spec-verified: architecture-design logs-and-apm.md lines 87-104 explicitly mandate three fields with the same value (for APM, logs, and mobile)
error.message	root cause	as designed
error.stack	source chain	as designed; the wrapper-embeds-source Display pattern causes cosmetic text duplication between entries
error.fingerprint	absent	expected for non-step errors (no step context)

Follow-up idea (recorded, not ticketed)

Emit a bare-code fingerprint from the Rust error! macro outside flow steps, matching what py-mando’s log_error now does (see the rework in BE-3482 pymando Branch Review). Would need one sentence added to the standard (logs-and-apm.md).

Related

• BE-3482 Datadog Logs and APM Conformance - the telemetry being deployed; MR !569 status note
• Mando Deployment Ceremony - the two-repo deploy ceremony this deploy instantiated (feature-branch variant on BOTH repos)
• BESS AM (BE-2262) - mando-bess-am - origin of the dev-only migrations behind the crash-loop
• be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 - the prior (inverse-direction) migration-drift crash-loop
• be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 - the same dangling-needs pipeline-creation failure class in the consumer repos
• BE-3482 pymando Branch Review - py-mando side of the bare-code fingerprint behavior
• Agent Context