Levandor

BE-3482 Datadog Logs and APM Conformance

12 min read

BE-3482 Datadog Logs and APM Conformance

Canonical status note for BE-3482 (ticket: “DataDog logs/APM core conformance”). Records the verified merge state (core work MERGED 2026-07-08 via MR !558), the 2026-07-13 continuation (the error.fingerprint activity extension plus the salvaged version-fallback panic fix, pushed and open as MR !569), the 2026-07-13 review AND same-day rework of the sibling pymando branch (rebuilt as feature/BE-3482-pymando-v2; see BE-3482 pymando Branch Review), the ranked list of still-open deferrals, and the field-verified tooling landmines recorded here (cargo fmt, RTK test rewrite).

For Agents

UPDATE 2026-07-22: the merge train COMPLETED; items (2), (6) and (7) below are superseded. !569 MERGED (53912a70), !570 MERGED (the pymando-v2 rework, squash fde2425e, merge 14dfb3ad, conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad on Nexus dev), !571 MERGED (BE-3541 Single Error Emission, develop tip 1c4c33a5). See Merge train completed (verified 2026-07-22). Dev now runs the throwaway build b7d15e2c; see Dev prototype verification (build b7d15e2c). Actionable state as of 2026-07-13: (1) do NOT trust the untracked repo handover HANDOVER-BE-3482.md (2026-07-02), it is stale on all state claims; (2) the fingerprint continuation is PUSHED and open as MR !569 (“feat: add activity component to error fingerprint”; branch feature/BE-3482-fingerprint-activity, worktree .worktrees/BE-3482-fingerprint, 2 commits: 6f541363 + the salvaged bc3727ba), awaiting review; (3) the orphan-commit salvage is DONE: 478cd993 was folded into the fingerprint branch as bc3727ba, and the old worktree .worktrees/logs-apm-conformance-core + local feature/BE-3482 branch are purely historical and archivable; (4) never run any form of cargo fmt, use rustfmt --edition 2021 <file> (see the tooling correction below); (5) on RTK machines never run cargo test ... -- --test-threads=1 through the shell hook, use rtk proxy cargo test ... and confirm nonzero passed totals (see the RTK landmine below); (6) the sibling pymando branch was reviewed AND reworked on 2026-07-13: the rework lives on feature/BE-3482-pymando-v2 (worktree .worktrees/BE-3482-pymando-v2, 15 title-only commits off develop tip 51d2b516, all gates green incl. pytest 120 passed / 1 skipped), NOT pushed, no MR yet; the old v1 branch/worktree are untouched as fallback; full outcome in BE-3482 pymando Branch Review; (7) the !569 telemetry is LIVE on dev since 2026-07-14 via the throwaway branch feature/BE-3482-dev-deploy (never to be merged) and was field-validated from the deploy’s own crash-loop; see Dev Deploy 2026-07-14 (error telemetry), which also records a STANDING dev crash-loop hazard for develop-based images (unmerged bess-am migrations in dev’s refinery history).

Merge train completed (verified 2026-07-22)

All three in-flight MRs of the error-telemetry train merged into develop between 2026-07-14 and 2026-07-22 (order per develop history, all shas verified against git 2026-07-22):

  1. MR !569 (fingerprint activity, feature/BE-3482-fingerprint-activity): merged by 2026-07-15 as squash b3f39605 “feat: add activity component to error fingerprint”, merge commit 53912a70 (the tip the AGENTS.md 07-15 refresh was built against).
  2. MR !570 (py-mando DD conformance, feature/BE-3482-pymando-v2): squash fde2425e “feat: pymando logging conformance”, merge commit 14dfb3ad. The conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad (sha suffix = the merge commit) is published to Nexus dev; BE-3613 moves the algo services onto it. The leftover branch feature/BE-3482-fingerprint-fallback is redundant since !570 merged; deletion still pending.
  3. MR !571 (BE-3541 Single Error Emission, feature/BE-3541): squash d1cf9c75 “refactor: emit step errors once by removing the logged-at-site path”, merge commit 1c4c33a5 = develop tip after !571. Closes deferral 1 below and the at-site-logged fingerprint gap.

In between, the unrelated BE-3643 merge (bf953ec8) landed (source of the pre-existing standalone cargo build -p mando_core uuid/serde failure), and develop has since moved to e2ff6b79 via the unrelated BE-3685 merge.

Next in the train: MR !578 open for BE-3656 (pipeline green, 4 reviewers, 0 comments as of 07-22), then BE-3657 (13 commits pushed @ 2baea68a, MR after !578 + explicit yes).

Merge state (verified 2026-07-13)

  • feature/BE-3482 was MERGED via MR !558 as squash ae6d1098 on 2026-07-08 (30 files, +402/-173, merge commit c2b7d401). The origin branch was deleted.
  • The untracked repo handover doc HANDOVER-BE-3482.md (dated 2026-07-02) is stale on all state claims. Ignore it for current state; this note supersedes it.
  • All 7 delivered field items were confirmed present on develop tip a01892f9.
  • What landed (per the round-2 guide verification of ae6d1098): error.message = root cause, error.stack = source chain, error.fingerprint = code|step_path, ErrorWithStepStatus.logged_at_site double-log guard with new()/logged() constructors. See [[Mando AGENTS.md Master Guide#error-redesign-partially-landed-commit-ae6d1098|Error redesign PARTIALLY landed (commit ae6d1098)]].
  • All deferrals remain open EXCEPT the fingerprint activity component, which the continuation work below closes (now open as MR !569, pending merge).

Orphan commit salvage (RESOLVED 2026-07-13)

  • Commit 478cd993 (Jul 9, “fall back to crate version when LOG_METADATA.version is None”) existed ONLY on the local feature/BE-3482 branch; it was never part of the merged MR.
  • Content: adds dd_formatter::app_version() and removes 3 panicking unwraps in mando-bess (api_docs.rs, flow/get_versions.rs, route/version_route.rs).
  • This is the fix for the known “develop panics mando-bess locally” issue.
  • Resolution: Andras chose to fold it into the fingerprint branch instead of a separate MR. Cherry-picked onto feature/BE-3482-fingerprint-activity as bc3727ba (“fix: fall back to crate version when LOG_METADATA.version is None”; the scope was dropped from the title, author preserved andras.lederer@alpiq.com).
  • The old feature/BE-3482 branch was NOT resurrected (it contains a merge commit and its shape was review-rejected).
  • The old worktree .worktrees/logs-apm-conformance-core and the local feature/BE-3482 branch are now purely historical and archivable.

Continuation: error.fingerprint activity component (2026-07-13, MR !569 open)

Implements the ACTIVITY component of the fingerprint per the authoritative standard (architecture-design repo, logs-and-apm.md): fingerprint = error.code + flow.step.path + specific activity (e.g. the Volue EMS data group id).

  • Branch feature/BE-3482-fingerprint-activity (worktree .worktrees/BE-3482-fingerprint), 2 title-only commits (author andras.lederer): 6f541363 (“feat: add activity component to error fingerprint”) and bc3727ba (the salvaged version-fallback fix, see the salvage section above). Pushed to origin and opened as MR !569 on 2026-07-13 (“feat: add activity component to error fingerprint”), awaiting review.

Design

  • Activity is a runtime value, so it rides ErrorWithStepStatus: new field activity: Option<String> plus a consuming with_activity() builder. All constructors default it to None, so behavior is bit-identical when absent.
  • A private fingerprint() helper produces code|path|activity, or code|path when activity is absent, with no trailing separator.
  • Reference wiring in mando-lib/src/service/volue/ems/ts_data_send.rs: every per-data-group error carries the data group name.
  • The real win: DA and AS schedule failures in the auction scheduling step previously collapsed into ONE Datadog error group; with the activity component they group separately.

Quality gates

  • 3 new capture tests.
  • Initial run (single-commit branch): clippy 0 warnings; mando_lib 254 passed, --test-threads=1.
  • Re-run on the 2-commit branch (after the bc3727ba cherry-pick): clippy exit 0; tests 387 passed / 0 failed across 22 suites, single-threaded.

Known limit (recorded honestly)

At-site-logged variants (e.g. VolueEmsError::ApiError) reach the step boundary as WARN with no fingerprint at all, so the activity is stored but not emitted for those paths. This stays broken until the single-emission migration lands (its own future ticket, top of the deferral ranking below). CLOSED 2026-07-22: the single-emission migration landed as MR !571 (BE-3541 Single Error Emission); fingerprint and http.* now emit for those paths too.

Sibling branch: feature/BE-3482-pymando reviewed and reworked (2026-07-13)

Read-only review of the 13-commit pymando DD-conformance branch (off develop@3024e20c, worktree .worktrees/BE-3482-pymando) against mando AGENTS.md, followed by a same-day executed rework. Full findings, gate transcripts, and the 9-item path to MR-ready live in BE-3482 pymando Branch Review; headline review facts:

  • Verdict: NEEDS REWORK BEFORE MR, though the conformance work itself is solid and the gates are genuinely green (clippy clean; 366 passed / 0 failed / 35 ignored via rtk proxy; mando_lib builds with --no-default-features AND with --features python, proving the new fmt_util needs no app feature).
  • Three blockers, about 30 minutes total: 13/13 scoped commit titles (banned per MR !322); 7 orphaned ContextVars plus dead set/reset lines in py-mando/python/py_mando/tracing.py after d528b71f removed their only reader; commits 86d7ffd8 + 34e749d5 are off-ticket AND byte-identical duplicates of bc3727ba on the fingerprint branch above.
  • Landing order: the fingerprint MR goes FIRST (based on the develop tip, unblocks the develop panic), then rebase pymando dropping the duplicated commits. Sharpest option also drops d0986230 (the fmt_util extraction: the entire develop conflict surface, off-ticket, only 2/3 done) for a zero-conflict 9-commit on-ticket MR, refiling the extraction as its own ticket.
  • The python pytest suite has not been re-verified since 2026-07-09 and must be re-run before MR (satisfied by the rework below: 120 passed / 1 skipped).
  • A third tooling landmine surfaced during the review: rustfmt --check --skip-children is unrecognized on stable and prints nothing (false green). Detailed in the review note; recorded in AGENTS.md section 10 + the landmines table, Mando AGENTS Guide (mirror) resynced.

Rework EXECUTED same day. Rebuilt from scratch as feature/BE-3482-pymando-v2 (worktree .worktrees/BE-3482-pymando-v2): 15 title-only, no-scope commits off origin/develop tip 51d2b516 (develop advanced twice during the session, including the BE-1595 Arrow Flight merge, which touches py-mando), zero cherry-pick conflicts, 10 files +374/-55. Commit selection: 9b0c4e70 KEPT (on-ticket, py-mando-only, independent of the drops); 86d7ffd8 + 34e749d5 + d0986230 DROPPED; notable surprise: fe78dc10 needed NO level_to_status workaround since the helper was already inline pre-extraction, so dropping d0986230 cost zero extra code. All review blockers and should-fixes executed (ContextVars deleted, tracing-serde removed, bare-code fingerprint outside trace scope with a new test, build_ddtags public, caplog + fixtures + genuine global restore). Also adds a NEW flow.step.system span field: implemented as ONE line in mando-lib/src/workflow/mod.rs from StepMetadata.system (the structured source of flow.step.connection, no string parsing); all 14 Wrapper steps verified declaring system: "Mando"; 4-case test matrix passes; scope tension flagged: one mando-lib commit (52b4d270) on an otherwise py-mando-only branch, trivially splittable. Gates: clippy exit 0 (zero findings in touched files); 434 passed / 0 failed / 35 ignored via rtk proxy; rustfmt zero branch-introduced hunks; pytest 120 passed / 1 skipped (baseline 114) including a runtime JSON conformance proof from the Rust stream. Two more tooling landmines recorded in AGENTS.md (scratch-dir rustfmt checks are false green + rustfmt writes into child files; develop’s committed Cargo.lock can be stale, so a lock hunk may be legitimate). v2 NOT pushed, no MR; the old v1 branch/worktree stay untouched as fallback. Full detail in BE-3482-pymando-v2. Superseded: v2 was subsequently pushed, opened as MR !570 and MERGED; see Merge train completed (verified 2026-07-22).

Dev deployment 2026-07-14 (field validation)

The !569 content (fingerprint activity + version-fallback fix) was hand-deployed to dev on 2026-07-14, while the MR still awaits review, via a throwaway branch: feature/BE-3482-dev-deploy = develop tip 51d2b516 + the two !569 cherry-picks + temporary CI edits, paired with the temporary IaC branch feature/error-hdl. Full mechanics, the bess-am migration crash-loop it hit (with the standing dev warning), and the terraform plan-reading lesson live in Dev Deploy 2026-07-14 (error telemetry).

The crash itself validated the telemetry live:

  • error.code == error.kind == error.type (spec-verified: architecture-design logs-and-apm.md lines 87-104 explicitly mandate three fields with the same value for APM/logs/mobile);
  • error.message = root cause; error.stack = source chain (the wrapper-embeds-source Display pattern causes cosmetic text duplication);
  • error.fingerprint absent, as expected for non-step errors.

Follow-up idea recorded there: emit a bare-code fingerprint from the Rust error! macro outside flow steps (matching py-mando’s log_error), which needs one sentence added to the standard.

Remaining deferrals (ranked)

  1. Single-emission migration: biggest win; unlocks fingerprint + http.* on at-site-logged errors. MERGED 2026-07 as MR !571; see BE-3541 Single Error Emission.
  2. error.* / http.* on trace spans. IMPLEMENTED on feature/BE-3656 via SpanEnrichmentLayer; open as MR !578 (pipeline green, awaiting review as of 2026-07-22); see BE-3656 APM Span Enrichment.
  3. Verify dd.trace_id correlation in Datadog. VERIFIED 2026-07 on the dev prototype build (sample trace 0dc8d42739a08ac0805808b2d14b6644); see Dev prototype verification (build b7d15e2c).
  4. flow.step.timeout_ms - still open (queued behind the BE-3657 later phases).

Tooling correction: cargo fmt does not scope

  • Field-verified during this work: cargo fmt -- <file> does NOT scope to the named file. It touched api_docs.rs, which was never named on the command line.
  • New prescription: format single files with rustfmt --edition 2021 <file>. Every form of cargo fmt is banned.
  • Propagated to the untracked mando AGENTS.md (sections 2.3, 10, 13.5 and the landmines table) and the vault mirror Mando AGENTS Guide (mirror) was resynced accordingly. This supersedes the earlier “scope with cargo fmt -- path/to/file.rs” advice in Mando AGENTS.md Master Guide and Agent Context (both amended).

Tooling landmine: the RTK hook silently filters out the whole test suite

  • Field-verified: the RTK shell hook rewrites cargo test --all-features --release -- --test-threads=1 so that the threads flag becomes a test-name filter: every test is filtered out and the command exits 0 in 0.00s.
  • Output signature of a gutted run: 0 passed, N filtered out.
  • Two gate runs were false-green this way before detection.
  • Correct invocation on RTK machines: rtk proxy cargo test --all-features --release -- --test-threads=1.
  • Always confirm the summed passed totals are nonzero; exit 0 alone proves nothing.
  • Adjacent shell pitfall: cargo test | grep | tail; echo $? reports tail’s exit status, not cargo’s.
  • Recorded in mando AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) already resynced.

Graph View

Backlinks