Levandor

Running mando Locally as a Host Process (Recipe)

3 min read

Recipe + friction log for running a locally built mando_bess binary as a HOST process against the mando-cli infra stack (postgres + wiremock + local-dd-agent). Assembled during the BE-3657 phase 3 local end-to-end verification (2026-07-23); the friction items are pre-existing repo issues, valuable for the whole team.

For Agents

The container path (full stack via mando up) is Local Docker Setup. THIS note is the host-process variant: binary on the host, infra in containers. Use it when you need to run an unmerged branch binary (not built into an image) against real infra.

The shape

mando up --datadog -p infra          # postgres + wiremock + local-dd-agent
cargo build --release                # in your worktree
set -a; source mando-p3.env; set +a  # env file lives OUTSIDE the repo (session scratchpad)
./target/release/mando_bess
  • Env assembled from .cargo/config.toml.example with overrides; keep the env file out of the repo (scratchpad, e.g. mando-p3.env).
  • Infra postgres creds (transactional DB): db bess_transactional_db, user postgres, password postgres, localhost:5432.
  • MANDO_PORT=8081 locally: wiremock owns 8080.
  • Point VOLUE_EMS_* / METIS_* / other adapter hosts at localhost:8080 (wiremock) for hermetic 404 error generation.
  • Set MANDO_ENVIRONMENT=dev-local-<user>: the embedded ddtags env wins over agent/envelope tags, see the epilogue.

Friction points (all field-verified 2026-07-23)

.cargo/config.toml.example is STALE

Env renames + a panic the example file does not know about

  • MANDO_FLOW_ACTIVE_VERSION is DEAD. The config now wants MANDO_SETUP_ACTIVE_VERSION (V1_4) + MANDO_SETUP_SCHEDULER_DISABLED (prefix MANDO_SETUP, config-crate style).
  • FINGRID_API_KEY must be NON-EMPTY: init panics via unwrap with “must contain at least one key”.

Postgres cert path is cwd-relative

The postgres cert loader reads <cwd>/certs/eu-central-1-bundle.pem. Running from a workspace root needs a certs symlink to mando-bess/certs. One was left in .worktrees/BE-3657-phase3 (untracked).

MIGRATION RACE: mando refinery vs mando-cli migrations container

Two writers apply the same migration set

In DuckDbPostgres mode mando’s own refinery races the mando-cli migrations container applying the same set. Symptom: idle-in-transaction lock pileup wedging boot (move_schema_objects blocked).

Recovery:

  1. pg_terminate_backend on the idle-in-transaction holder.
  2. Wipe schemas: drop public / data / bess_os cascade.
  3. Relaunch with a single writer.

MANDO_MODE=DuckDb skips mando-side pg migrations entirely and relies on the (stale) container set, which LACKS the setting table. Use DuckDbPostgres locally.

Datadog from a host process

The local dd agent tails ONLY mando-* containers (mando-cli-datadog-local-logs-2026-07-22), so a host process is invisible to it. Ship host-process error lines via DIRECT INTAKE to Datadog EU instead (the BE-3657 verification forwarded 12 lines under env:dev-local-levander service:bess-os-service-mando this way, after the ddtags rewrite below).

Epilogue (2026-07-23): the vanishing first forward

The first direct-intake forward of the mando error lines silently VANISHED despite 202-accepted responses.

  • Root cause: Datadog’s JSON-message parsing PROMOTES the embedded ddtags field from the log line itself. mando emits ddtags with env:local (from MANDO_ENVIRONMENT=local), which overrode the intake envelope’s env:dev-local-levander; the org evidently does not index env:local events at all, so accepted lines never appeared.
  • Fix that worked: rewrite the embedded ddtags to env:dev-local-levander before forwarding.
  • Isolation stake: the shared Alpiq org carries REAL env:prod telemetry under the same service:bess-os-service-mando name; the per-developer env tag is the ONLY isolation from production views.

Durable recommendation

Local mando runs should set MANDO_ENVIRONMENT=dev-local-<user> so the emitted ddtags align with the local agent’s env tag. Embedded ddtags win over host/env tags on JSON log lines, so even a containerized local mando would fight the agent’s tagging the same way (mando-cli-datadog-local-logs-2026-07-22).

Teardown

  • Stack: mando down.
  • Host process: pkill mando_bess.

Graph View

Backlinks