Levandor

BE-3656 APM Span Enrichment

7 min read

BE-3656 APM Span Enrichment

Implements deferral item 2 of BE-3482 Datadog Logs and APM Conformance (“error.* / http.* on trace spans”): copy the structured fields already carried on ERROR-level tracing events, plus per-request HTTP metadata, onto the active OpenTelemetry span as Datadog APM tags. Branch feature/BE-3656 (worktree .worktrees/BE-3656), 3 commits, originally stacked on the BE-3541 head dee24ad3; after !571 merged it was rebased zero-conflict onto the post-!571 develop 1c4c33a5 (new head 8f50ef2a) and went out as MR !578 (“feat: enrich trace spans with error and http attributes”), MERGED by Andras 2026-07-22: squash 46a9be1b, merge commit 4a0d297f = the new develop tip. Verified live on dev (error.* on errored step spans + Error Tracking pickup; http.* on GET/POST client child spans; log output byte-identical).

Review hardening landed in a0735ebb: bodies excluded from spans (deliberate spec deviation, raise with team), first-error-wins guard (marker extension, since set_attribute is append-only under the 128-attr SDK cap), u64 try_from, and shared OTel test helpers in test.rs.

For Agents

UPDATE 2026-07-22 afternoon: MR !578 MERGED (squash 46a9be1b, merge 4a0d297f = develop tip); see MR !578 (merged 2026-07-22). BE-3657 was rebased onto the merge the same day (13 commits replayed zero-conflict, head 2baea68a became 22ccae93, patch-id byte-identical) and gained the DD_SERVICE env fix ab8cfd11; its MR still waits on Andras’s explicit yes. The dev prototype has iterated to build b7d15e2c; current dev state lives in Dev prototype verification (build b7d15e2c). State as of 2026-07-15: branch feature/BE-3656 = BE-3541 head dee24ad3 + 2 commits (a64f9457, 6d4f4395), PUSHED, no MR. New file mando-lib/src/app/span_enrichment.rs holds SpanEnrichmentLayer. Gates green: 455 passed, only the 2 known TEST_PG_HOST archiver failures (mando-known-flaky-tests-2026-07-15 is a different flake). Log output byte-identical vs before (8/8 error-macro + 39/39 workflow capture tests). Spec + plan under repo docs/superpowers/{specs,plans}/2026-07-15-be3656-* (untracked, never commit). The four tracing/OTel gotchas below are reusable beyond this ticket.

MR !578 (merged 2026-07-22)

  • Opened after the 2026-07-16 session, once MR !571 (BE-3541 Single Error Emission) merged. Title: “feat: enrich trace spans with error and http attributes”.
  • Head 8f50ef2a = the same 3 commits, zero-conflict rebase onto the post-!571 develop 1c4c33a5. Pipeline GREEN.
  • Reviewers: krisztian.fekete1, gabor.nagy6, balint.budavoelgyi, jozsef.nagy1.
  • MERGED by Andras 2026-07-22 with 0 review comments: squash commit 46a9be1b “feat: enrich trace spans with error and http attributes”, merge commit 4a0d297f = the new develop tip.
  • Successor followed the same day: BE-3657 ran rebase --onto develop 4a0d297f, all 13 commits replayed zero-conflict (head 2baea68a became 22ccae93, patch-id verified byte-identical to the reviewed content; branch CI pipeline 2696324182 on 22ccae93 success), then gained the DD_SERVICE env fix ab8cfd11. Its MR still awaits Andras’s explicit yes.

Design

  • SpanEnrichmentLayer (mando-lib/src/app/span_enrichment.rs): a tracing_subscriber::Layer whose on_event copies error.* and http.* fields off ERROR-level tracing events onto the active OTel span as attributes (which become Datadog APM tags). It reads the active span via opentelemetry::Context::current().span(), NOT tracing::Span::current() (see gotcha 1).
  • instrumented_http_client: records http.method, http.url, http.status_code, http.version, and request/response content_length on the adapter span once per logical send, on the final outcome only (success, 4xx, or final failure after retries).
  • Bodies only via error events: request/response bodies are attached solely through error events, matching the “Only Error” column of the architecture-design logs-and-apm.md standard. No body tags on success spans.
  • No status writes: the layer never sets span Status; tracing-opentelemetry already does that on ERROR events (gotcha 3).

Quality gates

  • Log output verified byte-identical to pre-change: 8/8 error-macro capture tests + 39/39 workflow capture tests. Enrichment is additive to spans only; the log formatter is untouched.
  • Full suite: 455 passed, with only the 2 known TEST_PG_HOST archiver integration failures (CI-only, pre-existing per AGENTS.md section 9).
  • Clippy clean; scoped rustfmt on touched files.

Dev deploy

Superseded build

The prototype has since iterated: dev now runs 1.16.1-feat.2682437891.b7d15e2c (same throwaway branch, head b7d15e2c, IaC pin applied 2026-07-16). Current stack contents and the full Datadog verification live in Dev prototype verification (build b7d15e2c). The section below records the earlier d17227ac iteration.

As of 2026-07-16 ~02:10Z the dev environment runs a throwaway prototype build d17227ac (branch feature/BE-3482-dev-deploy, NEVER merge) that validates the BE-3657 target field shapes before the real work lands. Stack on top of the merged BE-3656 + BE-3541 span-enrichment base:

  • Full-path error codes prototype: error.code = FULL MODULE PATH.
  • Startup mock deep error (mando-bess/src/debug_error.rs): a one-shot 4-level error chain fired every boot inside a debug.mock_deep_error span, so the field shapes are always live to query.
  • error.errors / error.details as real JSON arrays: the mock emits an error.errors codes array + an error.details object array; the dd_formatter record_str path parses exactly those two field names into real JSON (targeted parse, not a general mechanism).
  • flow.step.system cherry-picked from MR !570 (7f3db1ed) onto the prototype.
  • IaC pin (optimization-universe-iac feature/error-hdl): 1.16.1-feat.2680023002.d17227ac, applied 2026-07-16 02:07Z, Apply complete.

Datadog array-attribute queries verified

Array element/wildcard/negation matching all confirmed working against error.errors / error.details:

  • element match: @error.errors:"<full code>"
  • wildcard: @error.errors:*MockAdapterError*
  • object-field: @error.details.code:*X*
  • numeric: @error.details.line:>10
  • negation: -@error.errors:*MockStepError*

Facets still to create (Andras click, one-time): @error.errors, @error.details.code, @error.fingerprint.

Arrays are log-side only

Datadog flattens array SPAN attributes to key.0 / key.1 tags, which breaks faceting. Arrays live on the log side only; spans keep the scalar error.code.

Tracing / OpenTelemetry Layer gotchas (reusable)

Field-verified facts about writing enrichment layers against tracing-opentelemetry 0.32 / opentelemetry_sdk 0.31, plus tooling landmines from this session. These generalize beyond BE-3656.

  1. tracing::Span::current() is EMPTY inside a Layer::on_event callback under a scoped subscriber. When the subscriber is installed with tracing::subscriber::with_default (as in tests), the scoped-dispatch reentrancy guard makes SCOPED_COUNT >= 1, so the nested get_default returns None and Span::current() yields an empty span inside on_event. Under a GLOBAL .init() subscriber it works. Fix for layer code: read opentelemetry::Context::current().span() instead. tracing-opentelemetry 0.32’s context_activation (default true) attaches the otel context on span enter/exit, so this path is correct in BOTH scoped tests and production. Note: outside layer callbacks (normal async code), Span::current() works fine even under scoped subscribers; the trap is specific to on_event.
  2. In-memory test exporter path changed. On opentelemetry_sdk 0.31 the exporter is opentelemetry_sdk::trace::InMemorySpanExporter (behind the testing feature), NOT opentelemetry_sdk::testing::trace::InMemorySpanExporter.
  3. Do not set span status in an enrichment layer. tracing-opentelemetry 0.32 already sets span Status::error("") on any ERROR-level event (layer.rs on_event, both OtelDataState arms). An enrichment layer setting status too is redundant and can clobber.
  4. OtelData.state is pub(crate) in 0.32. External code cannot manipulate the span builder directly. The only public write path is OpenTelemetrySpanExt (e.g. set_attribute). Caveats: set_attribute APPENDS, so duplicate keys are possible; Datadog keeps the last; the SDK span attribute limit defaults to 128.

Tooling landmines (this session)

  1. Bash cwd RESETS to the main checkout between calls. A cherry-pick intended for the dev-deploy worktree instead ran on bugfix/BE-3333 (the main checkout), producing phantom conflicts against the wrong base; recovered via cherry-pick --quit + restoring the single touched file. ALWAYS use git -C <worktree> (or cd <worktree> && ... in the SAME command) and absolute paths for edits. When conflicts look weird, first check WHICH repo/branch you are actually in.
  2. RTK proxy can garble grep/sed output into misleading results (apparent old-shaped code that was not actually there). Ground-truth with the Read tool on absolute paths, not grep/sed through the RTK hook.
  3. grep "test result" | tail -1 on cargo test output is a false green. It shows only the LAST test binary, which may be 0 passed / N filtered out. Always read ALL test-result lines and confirm the summed passed total is nonzero (compounds with the [[Mando AGENTS.md Master Guide|RTK --test-threads=1 filter trap]]).

Graph View

Backlinks