Levandor

mando-cli --datadog Local Log Shipping

6 min read

mando up --datadog ships a local mando up run’s error/log telemetry to the real Datadog EU UI, so BE-3482 Datadog Logs and APM Conformance dd_formatter JSON on stdout can be verified end-to-end from a laptop. Landed in the mando-cli working tree 2026-07-22, uncommitted by design (user commits manually). Committed since (tip c2becf1); the shipped guide docs/datadog-guide.md was verified accurate end to end on 2026-07-23 (dated section below).

For Agents

Purpose: no local→Datadog logs path existed before this. Dev ships logs via awslogs → CloudWatch → forwarder Lambda (see fr-region-missing-datadog-logs-2026-07-21); the ECS Datadog agent sidecar handles metrics/APM only. This feature adds a local Datadog agent container purely to validate the emitted log JSON in the real Datadog Logs explorer.

Usage

export DD_API_KEY=<key>
mando up --datadog        # flag must come BEFORE service args
  • Logs are tagged env:dev-local-<os-username> (sanitized to lowercase).
  • DD_SITE defaults to datadoghq.eu.
  • up prints a status row: datadog / env dev-local-<user>.
  • mando up without the flag, or mando down, stops shipping. up passes --remove-orphans, so a flag-off up removes the agent container.

Acceptance

CONFIRMED by the user 2026-07-24

Live run tested end to end — logs land in Datadog as designed. Acceptance closed.

Reference steps: export DD_API_KEY, run mando up --datadog, then in the Datadog Logs explorer filter env:dev-local-<you> → service:bess-os-service-mando events carrying error.code / error.fingerprint.

Why the service tag wins

The service tag comes from the log body ddtags, which overrides the agent’s infra enrichment. Verified: dd_formatter ddtags carries NO env entry, so the agent’s DD_ENV is authoritative for env while the body’s ddtags is authoritative for service.

Embedded ddtags env WINS when mando emits one (2026-07-23)

The “no env in ddtags” observation above only holds while MANDO_ENVIRONMENT puts no env entry into the body ddtags. Datadog’s JSON-message parsing PROMOTES the embedded ddtags from the log line itself: in the BE-3657 local e2e session a host-process mando with MANDO_ENVIRONMENT=local emitted ddtags env:local, which overrode the intake envelope’s env:dev-local-levander, and the org does not index env:local at all, so 202-accepted lines silently VANISHED. The same promotion would fight this agent’s DD_ENV tagging for a containerized local mando too.

Durable recommendation: local mando runs set MANDO_ENVIRONMENT=dev-local-<user> so the emitted ddtags align with the local agent’s env tag. This matters for isolation: the shared Alpiq org carries REAL env:prod telemetry under the same service:bess-os-service-mando name, so the per-developer env tag is the ONLY wall between local noise and production views. Details: Epilogue (2026-07-23): the vanishing first forward.

Mechanism

  • New infra template src/runtime/templates/infra/datadog.yml (agent image tag 7, container_name: local-dd-agent).
    • The literal name matters: the agent’s own log-include filter is name:^mando-.*, and naming the agent local-dd-agent guarantees the filter can’t match the agent container itself.
    • APM / process-agent disabled; mounts docker.sock + the containers dir.
  • Rendered to .infra/datadog.builtin.yaml only when --datadog is passed.
  • compose::assemble includes it existence-driven (optional) — deliberately NOT added to INFRA_BUILTIN_FILES, so a missing file never triggers a warning.
  • A flag-off up deletes .infra/datadog.builtin.yaml.
  • The up-side precheck resolve_datadog_env is the sole guard for a present-and-valid DD_API_KEY.

KEY GOTCHA: use ${DD_API_KEY:-}, never ${DD_API_KEY:?}

The template must default with :-, not error with :?. Compose interpolates env on every verb, so ${DD_API_KEY:?} empirically hard-fails mando down / mando logs in any shell that lacks the var while the agent file still exists. Verified live with docker compose 28.5.2 in both directions. The resolve_datadog_env precheck on the up path is the only place that should demand the key — the template must stay tolerant.

Side benefit: atomic writes unified

Extracted build_args::atomic_write_with_dir, unifying 3 write sites. override_gen’s write_override became atomic as a side benefit.

Process

  • Spec: docs/superpowers/specs/2026-07-22-datadog-local-logs-design.md
  • Plan: docs/superpowers/plans/2026-07-22-datadog-local-logs.md
  • Triple review flagged the :? footgun as Important; fixed and re-verified CLOSED.
  • Gate: build clean, clippy zero, 392 lib + 8 integration tests green.

2026-07-23 end-to-end guide test

Full test of docs/datadog-guide.md as shipped at mando-cli tip c2becf1. Verdict: guide accurate, every claim verified.

Verified

  • All four fail-fast paths print the exact documented messages with zero docker interaction: DD_API_KEY unset, DD_API_KEY empty, unresolvable OS username, unsanitizable OS username.
  • The agent renders only when the flag is passed; the results row datadog / env dev-local-levander is present.
  • The agent tails ONLY mando-* containers (DD_CONTAINER_INCLUDE_LOGS=name:^mando-.*, DD_CONTAINER_EXCLUDE_LOGS=.*); unrelated running containers are ignored.
  • Logs ship to the EU org (103 sent in the test run).
  • The teardown contract holds at both levels: mando up without the flag removes the agent via --remove-orphans, and mando down cleans everything.

Guide gap: the env tag lands late

Fresh logs appear WITHOUT the env tag at first

env:dev-local-<user> rides DD_ENV HOST metadata, and Datadog’s host-tag join takes up to about 10 minutes for a brand-new agent host. Freshly shipped logs are initially queryable by service: / container_name: only, then receive the env tag retroactively (about 8 minutes after agent start in this test). The guide’s Verifying section should say “wait a few minutes”, otherwise the env filter looks broken.

Hazard: stale pre-feature binary

A pre-feature mando binary silently starts a REAL mando up

A stale ~/.local/bin install reporting the SAME 0.4.0 version swallows --datadog as a service argument and runs a normal mando up. The guide’s “place the flag before service args” warning cannot protect against a binary that predates the flag. Andras rebuilt/reinstalled; suggested guide follow-up: a version-check note.

DD_API_KEY acquisition at Alpiq

The painful part of the test. Andras’s Datadog account (Alpiq Standard Operations role) lacks the “API Keys Read” permission, so the org API-keys page (app.datadoghq.eu/organization-settings/api-keys) is blocked. That dead end pushes people toward Personal Settings tokens (ddpat_ / ddapp_ prefixed), which do NOT work as DD_API_KEY: Datadog API keys are plain 32-hex, while application keys are 40-hex / ddapp_ and only pair with an API key for query APIs.

Working route: the deployed key lives in AWS Secrets Manager in the dev account (profile bessos-dev, region eu-central-1), secret DdApiKeySecret-pkeeEykkaqu3, referenced from optimization-universe-iac terraform/secret.tf via the GitLab CI variable DATADOG_API_KEY_SECRET_NAME:

aws sso login --profile bessos-dev
aws secretsmanager get-secret-value --secret-id DdApiKeySecret-pkeeEykkaqu3 --query SecretString --output text

See AWS CLI Access via Identity Center for the profile setup (and the rtk proxy aws output gotcha). Follow-up: request “API Keys Read” (or a personal org key) from the Datadog admin.

Programmatic verification

The Logs Search API (api.datadoghq.eu/api/v2/logs/events/search) works with that Secrets Manager API key paired with a ddapp_ application key from Personal Settings, for env:dev-local-levander verification queries.

Minor nit

The new mando-cli error strings use em dash characters, which goes against Andras’s writing conventions elsewhere; his call whether mando-cli cares.

Graph View

Backlinks