Levandor

FR (France) region services missing Datadog logs - root cause

5 min read

Root-cause analysis (2026-07-21, read-only AWS + IaC repo inspection) of why the FR (France) region BESS services produce no logs in Datadog: the extracted modules/bess_os terraform module creates the regional CloudWatch log groups but never wires the dd-logs-forwarder subscription filters, so logs stop at CloudWatch.

Context: the FR deployment

  • The France deployment is module "bess_os_fr" in optimization-universe-iac at terraform/bess_os_fr.tf (origin/develop).
  • Dev-only: count = var.environment == "dev" ? 1 : 0.
  • It instantiates the shared terraform/modules/bess_os module with region_code = "fr".
  • Runs in the bit-bessos-dev account 794038257734, eu-central-1 (same account/profile setup as AWS CLI Access via Identity Center).

Log shipping architecture in this stack

  • App containers use the awslogs driver, writing into CloudWatch log groups.
  • Datadog receives logs via CloudWatch subscription filters pointing at the datadog-forwarder Lambda.
  • Those subscription filters are created by the dd-logs-forwarder terraform module: git::https://gitlab.com/alpiq_cicd/terraform-modules/bit-tf-modules.git//dd-logs-forwarder?ref=v3.21.4
  • The Datadog agent sidecar in each ECS task handles metrics/APM/dogstatsd only, NOT container logs. An FR task looking “healthy” in APM while having zero logs is consistent with this split.

Root cause

The forwarding step was dropped when the module was extracted:

  • Top-level terraform/cloudwatch.tf creates the original log groups AND one dd-logs-forwarder module instantiation per group: datadog_mando_logs, datadog_optimization_logs, datadog_forecast_logs, datadog_dashboard_logs.
  • The extracted modules/bess_os/cloudwatch.tf creates the regional log groups (mando-fr, optimization-fr, forecast-fr, dashboard-fr, asset-simulator-fr) but contains no dd-logs-forwarder blocks at all.

For Agents

Failure class: module extraction dropped a side-resource. The log groups moved into modules/bess_os/cloudwatch.tf; the per-group dd-logs-forwarder instantiations stayed only in the top-level terraform/cloudwatch.tf and were never replicated into the module. Any future regional instantiation of modules/bess_os inherits the same gap until fixed in the module.

Live evidence (dev account, 2026-07-21)

  • All four original log groups /aws/ecs/bess-os/eu-central-1/dev/{mando,optimization,forecast,dashboard} each have a subscription filter to arn:aws:lambda:eu-central-1:794038257734:function:datadog-forwarder.
  • The four -fr counterparts have zero subscription filters.
  • The FR services are actively writing to CloudWatch: mando-fr and forecast-fr last events were ~27 minutes old at check time. So logs exist, they just never leave CloudWatch.

Fix direction (not applied)

Add dd-logs-forwarder module instantiations inside modules/bess_os/cloudwatch.tf for each regional log group: mando, optimization, forecast, dashboard, plus asset_simulator when enabled, mirroring the top-level terraform/cloudwatch.tf blocks.

Sibling exemplar

modules/bess_am/ecs.tf (the bess_am module) has its own “Datadog Log Forwarding” section: a module that DID carry its forwarding wiring along. Useful shape to copy when patching modules/bess_os.

Deployment of the fix follows the usual IaC ceremony (plan on develop, manual terraform_apply:dev), see Mando Deployment Ceremony.

Fix implementation and feature-branch deploy (2026-07-21)

  • Fix implemented on branch bugfix/fr-datadog-log-forwarders of optimization-universe-iac (worktree .worktrees/fr-dd-forwarders), commit c169afc fix: add datadog log forwarders to bess_os module: seven dd-logs-forwarder blocks appended to terraform/modules/bess_os/cloudwatch.tf, the asset_simulator one count-gated on var.asset_simulator_enabled.
  • Verified before deploy: the dd-logs-forwarder v3.21.4 source creates exactly one aws_cloudwatch_log_subscription_filter per instantiation targeting the account’s datadog-forwarder Lambda; the Lambda resource policy allows logs.amazonaws.com for log-group:*, so no per-group permission is needed; the customer_portal module proves in-module instantiation works live.
  • New ceremony learned: the IaC repo’s terraform_apply:dev job is gated only: [develop]; to deploy a feature branch to dev, temporarily add the branch regex to that only: list (commit 8592eeb ci: enable dev apply on fr datadog forwarders branch, one line in .gitlab-ci.yml). Apply remains when: manual. The CI commit must be dropped before merge, same as mando AGENTS.md section 14.3.

Known drift to watch in the plan

The live FR stack runs a 1.16.1-feat mando image and an asset-simulator container, while develop pins other versions and sets asset_simulator_enabled = false for fr; a develop-based apply may revert those.

  • Module source fetched read-only via SSH clone of git@gitlab.com:alpiq_cicd/terraform-modules/bit-tf-modules.git (https clone fails auth; ssh works).

Follow-up: FR logs show unsuffixed service name (2026-07-22)

After the forwarder fix, FR logs DO arrive in Datadog, but tagged service:bess-os-service-mando instead of bess-os-service-mando-fr.

  • Investigated live: the log group AWS tag, container DD_SERVICE env, docker labels, ECS resource tags, and forwarder lambda tags are ALL correctly -fr-suffixed. Nothing in the infra is wrong.
  • Each FR log event in Datadog carries TWO service tags: service:bess-os-service-mando (wins) and service:bess-os-service-mando-fr (added by forwarder log-group tag enrichment, DD_ENRICH_CLOUDWATCH_TAGS).

Root cause is in mando, not IaC

mando embeds a ddtags field in every JSON log line via dd_formatter, built from the hardcoded constant pub(crate) const DD_SERVICE: &'static str = "bess-os-service-mando"; at mando-lib/src/app/mod.rs:38 (develop). A unit test (dd_formatter.rs ddtags_include_unified_service_tags) even asserts the hardcoded value. The embedded ddtags service wins over forwarder enrichment in Datadog.

Fix direction (mando repo): read the DD_SERVICE env var at init in dd_formatter/init_log, keeping the constant as fallback; optionally do the same for the hardcoded global::meter("bess-os-service-mando") names in service_base.rs:120-132.

Mando-side fix implemented (2026-07-22)

Shipped as commit ab8cfd11 “fix: read dd service name from env” on feature/BE-3657 (Andras asked to include it in the upcoming MR; full detail in DD_SERVICE env fix (commit ab8cfd11, 2026-07-22)):

  • resolve_dd_service() + a OnceLock-cached dd_service() accessor in mando-lib/src/app/mod.rs (app feature gated); hardcoded value stays as fallback, also on empty string.
  • All 6 hardcode sites routed through it: OTel Resource service name (get_tagging), ddtags in dd_formatter.rs, and 4 raw global::meter("bess-os-service-mando") literals in service_base.rs (3x) + util/http_client_trace.rs that had bypassed the old const.
  • Deployment-neutral on the main stack: terraform/bess_os_ecs.tf:218 already sets DD_SERVICE = "bess-os-service-mando" on the mando container, so behavior only changes where the env differs. The FR stack’s -fr-suffixed value finally wins once a build carrying the fix deploys there.
  • Not yet merged: rides the BE-3657 MR, which waits on Andras’s explicit yes.

Debugging technique

Raw CloudWatch event inspection via aws logs get-log-events is the fastest way to see what mando actually emits, including the embedded ddtags line.

Graph View

Backlinks