Levandor

Zero trace correlation on Rust-origin logs inside Python containers

4 min read

Rust tracing events emitted from inside the Python algo containers carry no dd.trace_id / dd.span_id and no flow context, because py-mando ships its own JSON log formatter that never writes trace fields, and no ddtrace span context is ever bridged from Python into the Rust side. Verified 2026-09-02 by reading code at mando origin/develop tip b5770766.

Symptoms

Observed on bess-os-algo-forecast-fr (dev, FR region). A log line has:

  • asctime
  • logger.file: mando-lib/src/service_base.rs:937
  • logger.line

and is missing:

  • logger.name
  • thread_name
  • dd.trace_id / dd.span_id
  • flow.exec_id / flow.step.path

Net effect: the line is orphaned in Datadog. It cannot be correlated to an APM trace or to a flow execution.

Root cause: two independent Rust log formatters

mando-lib/src/app/dd_formatter.rs (mando service)py-mando/src/log_formatter.rs (Python containers)
timestamp keytimestampasctime
statusyesyes
logger.file / logger.lineyesyes
logger.name (meta.target())yesno
logger.thread_nameyesno
dd.span_id / dd.trace_idyesno
ddtagsyesyes
span fields / event fieldsyesyes

git grep 'dd.trace_id|span_id|DD_LOGS_INJECTION' over py-mando/ returns nothing: the trace fields are simply never written on that path.

Why a subset was reimplemented rather than reused

Two structural reasons the py-mando formatter is a thinner copy instead of a caller of dd_formatter:

  1. struct TraceInfo (dd_formatter.rs:62) and fn lookup_trace_info (dd_formatter.rs:79) are private.
  2. mando-lib/src/lib.rs:2 gates pub mod app behind the app feature, which py-mando does not enable (it enables python).

These are NOT the thing to undo

Do not read the two blockers as “unwall the helper and reuse it”. That fix direction was investigated and ruled out (2026-09-02, verified at origin/develop tip f7fb74ef) - see Ruled out: moving TraceInfo into mando-core. They explain the shape of the current code, nothing more.

For Agents

set_ddtags IS wired (py-mando/src/lib.rs:129), so ddtags do flow through from the Python side. The missing piece is specifically trace correlation, not tagging. Do not chase the ddtags path when debugging this.

Second, independent gap: flow context never reaches Rust events

TraceFilter in py-mando/python/py_mando/tracing.py injects flow.exec_id / flow.step.path from Python contextvars, but it is a logging.Filter and therefore only runs on Python logging records. Rust tracing events go straight to stdout via the Rust formatter, bypassing Python logging entirely, so the contextvars set by run_with_trace never reach them.

This is a separate defect from the dd.trace_id gap and is not fixed by fixing the formatter.

Ruled out: moving TraceInfo into mando-core

The obvious-looking fix (“move TraceInfo + lookup_trace_info into mando-core, make them pub, call them from py-mando’s formatter”) does not work. Verified at mando origin/develop tip f7fb74ef.

  1. It is an OTel helper, not a generic one. lookup_trace_info resolves trace ids via span_ref.extensions().get::<OtelData>(), i.e. tracing_opentelemetry::OtelData (dd_formatter.rs imports opentelemetry::trace::{SpanId, TraceId} at line 4 and tracing_opentelemetry::OtelData at line 11).
  2. It would violate the mando-core weight rule. py-mando/Cargo.toml declares only tracing.workspace = true and tracing-subscriber.workspace = true - no opentelemetry, no tracing-opentelemetry. Moving the helper drags both OTel crates into the crate that AGENTS.md section 1 requires to stay lightweight, because mando-core is compiled into the Python extension.
  3. Decisive: it would still return None. py-mando/src/lib.rs init() builds the subscriber as tracing_subscriber::registry().with(EnvFilter).with(tracing_subscriber::fmt::layer().json().event_format(JsonFormatter)). There is no OpenTelemetry layer, so nothing ever populates OtelData in any span’s extensions. The helper would compile and return nothing.
  4. Even with an OTel layer it would be the wrong ids. They would be Rust-side trace ids with no relationship to the Python ddtrace spans that run_with_trace creates via tracer.trace(context). Datadog correlation requires the ddtrace trace_id/span_id of the active Python span.

Fix direction

Split in two, because the metadata half and the correlation half have completely different risk profiles.

B1: formatter parity (small, no design risk)

Add logger.name (from meta.target()) and logger.thread_name to py-mando/src/log_formatter.rs, matching mando-lib/src/app/dd_formatter.rs. This closes the metadata half of the gap and is independent of trace correlation - it can ship on its own.

B2: real trace correlation (own ticket)

Actual dd.trace_id / dd.span_id correlation requires bridging the active ddtrace span context from Python into Rust so the Rust formatter can stamp it.

Natural shape: mirror the existing set_ddtags mechanism (py-mando/src/lib.rs:129 → LOG_DDTAGS OnceLock in log_formatter.rs), except it must be per-call rather than set-once. run_with_trace in py_mando/tracing.py already holds the span and could push span.trace_id / span.span_id across, clearing on exit.

Known ceiling to design around

A naive global would be wrong under concurrent flows in one process. The storage must be task- or thread-scoped, or the containers’ single-flow-at-a-time assumption must be stated explicitly as a precondition.

Reusable precedent

mando-lib already has a TASK_CONTEXT tokio task-local (used in workflow/mod.rs) - likely the right shape for the per-call storage.

Also out of scope of both: flow-level correlation across the Rust/Python boundary (the TraceFilter / contextvars gap above) is a further, larger design question.

Graph View

Backlinks