Levandor

BE-4014 Datadog Error-Rendering Test Harness

6 min read

BE-4014 Datadog Error-Rendering Test Harness

A small, develop-based deliverable: a public test helper that lets you unit-test “what would this error look like on Datadog” by driving the REAL error! macro through the REAL DatadogFormatter and asserting the captured DD JSON. Bundled with it (the genuinely reusable part) is a survey of develop’s DD error-emission contract, now pinned by the harness’s worked-example tests.

For Agents

Deliverable branch feature/BE-4014 @ 9a8a7538, PUSHED, NOT yet an MR (awaiting Andras’s explicit yes). Renamed from the provisional feature/dd-error-test-harness; the worktree dir .worktrees/dd-test-harness is historical. This is independent of the error-stack train (BE-3657 error_stack Adoption MR !585 / BE-4000). The contract survey below is develop’s contract; the error-stack branches change/extend it (see the caveats inline). Update 2026-07-29: the harness gained a golden/snapshot MATCHER (Andras-requested A/B style); tip eac5fb85 → 9a8a7538 - see Golden JSON matcher (added 2026-07-29).

The deliverable

Motivation: a teammate needed a way to unit-test DD error rendering, generate an error, assert its rendered Datadog JSON, without standing up a live pipeline.

What it is: a pub test helper

render_dd(|| error!(...)) -> Vec<serde_json::Value>   // all captured DD JSON lines
render_dd_one(|| error!(...)) -> serde_json::Value    // asserts exactly one line

in mando_lib::app::dd_formatter::test_support. It drives the REAL error! macro through the REAL DatadogFormatter and returns the captured DD JSON. It promotes the previously-private capture harness that lived in dd_formatter.rs’s own test module, so any crate/test can now assert rendered DD output instead of re-implementing capture.

Feature combo: consumers build with --features app,test-util. The test-util feature is new on mando-lib and is defined as test-util = ["mando_core/test-util"] (it forwards to mando-core’s existing tracing/DataFrame test helpers).

Worked examples: 9 example tests in a #[cfg(all(test, feature = "test-util"))] mod render_dd_examples serve as the copy-paste templates for the common assertions (report arm, plain arm, http arm, deepest-cause message, etc).

Run it:

cargo test -p mando_lib --features app,test-util

Golden JSON matcher (added 2026-07-29)

Second capability on the branch (tip eac5fb85 → 9a8a7538): a golden/snapshot MATCHER so a test can assert a STATIC expected error.* JSON against the generated DD JSON (Andras’s requested A/B style - write the expected shape once, compare). Two pub helpers alongside render_dd in mando_lib::app::dd_formatter::test_support:

assert_dd_matches(generated, expected)      // compare a captured DD Value against a golden Value
assert_error_renders(body, expected)        // render an error!(...) closure and match in one call

Match semantics (deliberately asymmetric - the golden is a partial spec):

  • Objects = recursive SUBSET: every key in expected must be present and match; EXTRA keys in generated are ignored - so you omit volatile fields (file, line) from the golden and they never break it.
  • Arrays = same-length, element-wise: length must match, then each element recurses.
  • Strings = exact, OR a leading * means ends_with: "*VolueEmsError::ApiError" matches any full module-path code ending in that segment - a grep affordance for the long full-path codes.
  • Scalars (numbers/bools/null) = exact.
  • On mismatch it panics with the JSON path to the offending node (e.g. error.details[0].code), so a failure points straight at the diverging field.

Because a positive golden cannot assert a field is ABSENT, the pre-existing field-by-field absence tests are KEPT as complementary (e.g. “the report arm emits no fingerprint”); the golden examples do not replace them.

Shipped tests: the matcher is self-tested with 8 tests including should_panic negatives (subset pass, extra-key-ignored, array-length-mismatch, *-suffix pass/fail, wrong-scalar panic, path-in-panic-message). Three golden json!({...}) example tests were added as the NEW copy-paste template for “assert the whole DD shape at once” (report arm, plain arm, http arm), sitting beside the 9 field-assertion worked examples. Full suite at 9a8a7538: 39 passed / 0 failed under --features app,test-util.

Develop DD error-emission contract (surveyed + now test-pinned)

The useful knowledge. This is what the error! macro + DatadogFormatter emit on develop. The harness pins these shapes; the BE-3657 error_stack Adoption branches (!585 / BE-4000) change or extend several of them, flagged inline.

Per-arm field shapes

error! armEmitsDoes NOT emit
error!(report = ...)error.errors[] (full-path codes, outermost-first); error.details[] = {code, file, line, message} per level; error.code == error.kind == error.type == the outermost full-path code; error.message = deepest causeerror.fingerprint (none on the report arm)
error!(err) plain arma bare error.fingerprint == error.codeerror.errors / error.details
error!(err, http_context = ...) http armsame bare error.fingerprint == error.code, plus the http.* fieldserror.errors / error.details

Cross-cutting facts

  • The {code}|{step_path}|{activity} fingerprint is NOT the macro’s. The plain/http arms emit only a bare error.fingerprint equal to error.code. The structured {code}|{step_path}|{activity} fingerprint is produced by StepResult::log in the workflow layer (see BE-3541 Single Error Emission), not by error!.
  • No redaction at the rendering layer. The macro and DatadogFormatter emit http.*.body verbatim: None is omitted, Some(...) is rendered including any secrets. The no-leak guarantee lives entirely in caller code passing None, not in the formatter. Do not assume the render layer scrubs anything.
  • error.details has NO attributes object on develop. The ErrorAttr / attributes per-level object is branch-only; it arrives with the error-stack MRs (BE-3657 error_stack Adoption), not on develop.

Separate concern, DIAGNOSED not fixed

Same session, unrelated to BE-4014. This is someone else’s develop feature (“pymando logging conformance”, merge fde2425e); recorded here only because it surfaced alongside the DD-rendering work. Diagnosed only, NOT fixed.

develop’s Windows py-mando CI job is red on 3 tests in test_dd_conformance.py:

  • test_log_error_uses_deepest_cause
  • test_log_error_without_step_context_omits_path
  • test_log_error_extra_does_not_override_error_fields

Symptom: IndexError on empty caplog.records.

Root cause = test isolation, not a logic bug. log_error (tracing.py) is a pure logger.error with no global mutation, so an empty caplog means caplog is not capturing because global logging state leaked. dictConfig defaults disable_existing_loggers=True in test_logging_conf_emits_conformant_keys and in tracing.load_log_config; the logger_state fixture only restores root handlers/level, not per-logger disabled / propagate. The first caplog test passes and the next 3 fail: the classic leak signature.

Fix direction (not applied): set disable_existing_loggers=False, and/or reset the dd-conformance-error logger in the error_log fixture.

Graph View

Backlinks