Levandor

BE-3657 error_stack Adoption

24 min read

BE-3657 error_stack Adoption

Locked design decision to adopt the error_stack crate (=0.8.0) wholesale as mando’s error-handling substrate, replacing the ErrorCode-derive direction. Follow-on to BE-3656 APM Span Enrichment. Spec’d 2026-07-15/16; phases 1+2 MERGED 2026-07-23 via MR !580 (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 (plumbing + adapters) IN REVIEW since 2026-07-23 evening as MR !585 (feature/BE-3657-phase3 → develop, now @ f771805e, ~27 commits - see MR !585 opened (2026-07-23 evening)); review scope includes the 2026-07-23 short error.code decision (see Short error.code decision (2026-07-23)), the 2026-07-24 Caused-by error.stack renderer that REVISES locked decision 3 (see error.stack Caused-by renderer decision (2026-07-24)), and the 2026-07-24 error-authoring DX overhaul (report vocabulary, short error.details codes, native report signatures via the MandoReport newtype; the next MR, a derive(MandoReport) proc macro, is already spec’d - see Error-authoring DX overhaul (2026-07-24)). The dev environment runs a throwaway build carrying the phase 1+2 stack, field shapes verified in Datadog (see the dev prototype section).

For Agents

Status 2026-07-24: MR !580 (phases 1+2) MERGED (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 IN REVIEW as MR !585 “feat: adopt error stack across all adapters” (feature/BE-3657-phase3 → develop, head f771805e, ~27 commits, pushed; reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1; branch pipeline was green at a58df341, local gates at f771805e: 580/0 workspace lib tests + clippy clean) - see MR !585 opened (2026-07-23 evening). In review scope: the 2026-07-23 SHORT error.code decision (commit a58df341: Enum::Variant on error.code only) - see Short error.code decision (2026-07-23) - the 2026-07-24 Caused-by error.stack renderer (commit 1f9cd611: custom “Caused by” chain renderer replaces the crate’s Debug tree, REVISES locked decision 3) - see error.stack Caused-by renderer decision (2026-07-24) - and the 2026-07-24 error-authoring DX OVERHAUL (commits f5beabd4 report vocabulary / 4e519269 short error.details codes / f771805e native report signatures + MandoReport newtype) - see Error-authoring DX overhaul (2026-07-24). Execution history: Phase 3 executed (2026-07-22 evening through 2026-07-23). Worktree .worktrees/BE-3657-phase3. Specs: docs/superpowers/specs/2026-07-15-error-stack-adoption-design.md (phases 1+2, includes the flipped template + annotation direction) + docs/superpowers/specs/2026-07-22-be3657-phase3-adapters-design.md (phase 3; now records the walled-towers formal exception) + docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md (next MR: derive(MandoReport) From-impl generation). Plans: docs/superpowers/plans/2026-07-15-be3657-error-stack-phase1.md + 2026-07-16-be3657-error-stack-phase2.md + 2026-07-22-be3657-error-stack-phase3.md. All untracked, never commit. Remaining: the spec’d derive(MandoReport) MR (next up), phase 4 (#[error_meta(...)] annotations, Andras’s own work) + the follow-up ticket bundle in the execution section. Phase 3 telemetry VERIFIED LOCALLY end to end 2026-07-23 from the real runtime (binary b394e0d4, a rebase-descendant; the short-code commit a58df341 and the Caused-by-renderer commit 1f9cd611 also verified live) and the DX batch is verified live 2026-07-24 through a CONTINUOUS forwarder to Datadog (env:dev-local-levander, ~5s latency; complete field contract confirmed: code short, errors full, details codes short per level, fingerprint full, Caused-by stack, nested http on 5xx paths) - see Local end-to-end verification (2026-07-23) + Local live stream operational (2026-07-24); host-process run recipe: mando-local-host-run-recipe-2026-07-23.

Locked decisions (2026-07-15/16, Andras)

  1. Full error_stack (=0.8.0) adoption over an ErrorCode-derive macro. Guiding principle: “leave what we can to error_stack” - mando contributes ONLY:
    • the ErrorCode attachment (full-path code captured statically in a CodedExt helper at each change_context / new site),
    • extraction functions, and
    • an error!(report = ...) macro arm. Everything else (chaining, source walking, rendering) is the crate’s job. Amended 2026-07-24: the authoring vocabulary changed - CodedExt / new_coded / change_context_coded are DELETED workspace-wide, replaced by ToReport::into_report / Result::reported() / .wrap() / .attr(k, v); the principle itself stands - see Error-authoring DX overhaul (2026-07-24).
  2. Field contract (single outermost identity across all three):
    • error.code == error.kind == error.type == the outermost full-path code.
    • error.fingerprint uses the full-path code.
    • error.errors = array of codes.
    • error.details = array of {code, file, line, message} objects.
    • Amended 2026-07-23: error.code alone now carries the SHORT readable form (Enum::Variant; bare struct name for variantless types); kind/type/errors/details codes/fingerprint stay full module path - see Short error.code decision (2026-07-23).
    • Amended again 2026-07-24: error.details[].code joins the short form (the details panel is the human panel; each entry’s file:line carries the precision); kind/type/errors/fingerprint stay full path - see Error-authoring DX overhaul (2026-07-24).
  3. Tree rendering = the crate’s own output, with codes attached as sub-lines (attachment style), NOT a prefix style. Accepted trade-off. REVISED 2026-07-24: after seeing the tree in the real Datadog UI, the crate’s Debug tree is dropped for a custom prefix-style “Caused by” chain renderer - see error.stack Caused-by renderer decision (2026-07-24).
  4. Boundary message template FLIPPED to STATIC-FIRST on 2026-07-16: "{static}: {outermost}[: {chain}]". Supersedes the earlier outermost-first order ("{outermost}: {static}: {chain}") recorded on 07-15/16; the flip is in the authoritative spec and deployed in the dev prototype build.
  5. Phase-2+ annotation direction (Andras’s own work once the phase-1 plumbing exists, must not be precluded by phase 1): per-variant #[error_meta(event_type, resolution step, business_message, description)] feeding error.details enrichment plus resolution-doc codegen. error_stack attachments are open-ended, so phase 1 does not block this.
  6. Bodies never on spans (shared with BE-3656): request/response bodies attach only via error events, a deliberate deviation from the logs-and-apm.md spec table, STILL to be raised with the team.
  7. error.errors / error.details excluded from span tags: the arrays live on the log side only (Datadog flattens array span attributes to key.0/key.1, breaking facets); spans keep the scalar error.code.
  8. First-error-wins span enrichment guard (from BE-3656): a marker extension prevents later errors overwriting span error tags, since set_attribute appends under the 128-attr SDK cap.
  9. HTTP fields nested as http.request.* / http.response.* (axum semconv-aligned; collision investigated, none).

Build state (2026-07-22)

Phases 1+2 are done and pushed. Morning state:

  • 13 commits @ 2baea68a atop the BE-3656 head 8f50ef2a (feature/BE-3657 on origin, ls-remote verified). Contains BOTH phases; suggested MR reflects that.
  • Rebased 2026-07-21. During the rebase, 3 develop-inherited bare-code fallback tests were aligned to full-path expectations and squashed into the commit “feat: use full module path in error code”.
  • Gates green via the workspace build graph (mando_core 94/0 through the workspace). Standalone cargo build -p mando_core fails on a PRE-EXISTING uuid/serde compile issue inherited from the BE-3643 merge (bf953ec8), not caused by this branch.
  • Later phases remain: phase 3 (planned 2026-07-22 evening as plumbing + adapters, superseding the earlier “workspace breadth” framing - see Phase 3 planned (2026-07-22 evening); the py-mando formatter parse-hook port and the tuple/transparent restructuring are IN it) and phase 4 (the #[error_meta(...)] annotations, Andras’s own work).

Afternoon 2026-07-22: rebased onto develop after !578 merged

  • MR !578 merged (squash 46a9be1b, merge 4a0d297f = new develop tip), so the pending rebase --onto develop happened: all 13 commits replayed zero-conflict, head 2baea68a became 22ccae93, patch-id verified byte-identical to the reviewed content.
  • Branch CI pipeline 2696324182 on 22ccae93: success.
  • Then commit ab8cfd11 “fix: read dd service name from env” was appended (Andras asked to sneak it into the upcoming MR), making the branch 14 commits @ ab8cfd11, pushed and ls-remote verified. Details in DD_SERVICE env fix (commit ab8cfd11, 2026-07-22).
  • Local gates green after the append: clippy clean, 516 tests passed / 0 failed at lib scope (--lib = CI parity, see mando-ci-lib-only-test-gate-2026-07-22).
  • MR ceremony: nothing left but Andras’s explicit yes.

Phase 3 planned (2026-07-22 evening)

Phase 3 design approved by Andras 2026-07-22 (evening session). Kept as the planning record; execution ran the same evening through 2026-07-23, see Phase 3 executed (2026-07-22 evening through 2026-07-23).

Artifacts (untracked, never commit): spec docs/superpowers/specs/2026-07-22-be3657-phase3-adapters-design.md; plan docs/superpowers/plans/2026-07-22-be3657-error-stack-phase3.md (15 tasks).

Locked phase 3 decisions

  1. Scope = “plumbing + adapters”. Rejected alternatives: plumbing-only; full workspace breadth.
  2. Mechanics = seam replication per the Volue pilot. Rejected alternatives: native Report signatures; boundary-only wrapping. Pilot artifacts already on the base branch: retrieve_ts_data_reported, send_request_reported, volue_error_from_report in adapter/volue/ems/ts_data_retrieve.rs. REVISED 2026-07-24: the seam pattern was eliminated on the same branch in favor of the once-rejected native report signatures, made viable by the MandoReport newtype’s ? auto-lift (commit f771805e) - see Error-authoring DX overhaul (2026-07-24).
  3. Ticket = keep riding BE-3657: branch feature/BE-3657-phase3 off develop AFTER !580 merges; one phase = one MR.

Conformance pass against the team spec

The design was tightened against Balazs’s team spec (architecture-design repo, doc/content/doc/ops/logs-and-apm.md, fetched from main 2026-07-22):

  1. Line 157 is absolute: ALL tuple/transparent variants in migrated enums get restructured.
  2. error.details gains the spec’s attributes field (lines 150-155) via a new mando-core ErrorAttr attachment + extraction; this lands BEFORE any adapter migrates.
  3. Line 159 http roll-up: implemented as an outermost-wins pick of the report’s HttpContext attachment; with_report fills http_context when unset. The spec’s “force with a macro” wording is interpreted positionally and goes on the pending team-sentences list.
  4. Auth provider migrations carry a hard sensitive-data rule (lines 121-124): no bodies, no tokens; method/url/status only.

Adapter inventory and order (grounded on the branch)

  1. Auth providers FIRST: alpiq/authentication + the Volue EMS/ATP auth providers (sensitive-data rule above applies).
  2. volue/ems completion: spot + send join the ts_data_retrieve pilot.
  3. volue/atp.
  4. alpiq/ebs: SMB, so NO http roll-up; share/path ErrorAttr instead.
  5. alpiq mdr / opl / data_platform / et_3000.
  6. fingrid.
  7. alpiq/metis.
  8. mando group: rest / algo / simulator / ms_teams / microsoft.
  9. alpiq/position_manager.

Flagged defaults (Andras can veto before execution)

Two vetoable defaults baked into the plan

  1. adapter/mando/archiver (+ its postgres submodule) DEFERRED to the repos phase: data-plane, separate CI harness, no step boundary.
  2. The startup mock deep error gets PORTED to mainline mando-bess (it exists only on the throwaway dev-deploy branch) behind MANDO_DEBUG_MOCK_ERROR, default-off; one IaC dev env line at deploy time; droppable if the team objects.

New mando-core surface (phase 3 task 1)

  • pub struct ErrorAttr(pub String, pub serde_json::Value) attachment.
  • error_attr_value: one-level-deeper rendering for error-typed attributes.
  • report_first_attachment: outermost-wins generic pick.
  • report_details entries gain attributes.
  • A shared error_json_field_value fn unifies the error.errors/error.details JSON parsing across dd_formatter and both python package log formatters (py-mando/src/log_formatter.rs, py-mando-simulation/src/log_formatter.rs).

Phase 3 executed (2026-07-22 evening through 2026-07-23)

Phase 3 ran subagent-driven on branch feature/BE-3657-phase3: 14 tasks + a final whole-branch review, every task under an adversarial spec + quality review. The branch was initially stacked on the in-review !580 head ab8cfd11; after !580 merged, rebase --onto develop replayed with zero conflicts, patch-id byte-identical. 20 commits at execution end, review HEAD e0184bc3, PUSHED (ls-remote verified). Update 2026-07-23 evening: 22 commits @ a58df341, IN REVIEW as MR !585 - see MR !585 opened (2026-07-23 evening). Neither flagged default was vetoed: the archiver seam migration stayed deferred per the plan (the transparent sweep still covered adapter/mando/archiver), and the mock port shipped env-gated.

Delivered

  • mando-core surface: ErrorAttr attachment + per-level attributes on error.details entries; error_attr_value one-level nesting for error-typed attribute values; report_first_attachment outermost-wins generic pick; with_report http roll-up (fills http_context from the report’s outermost HttpContext when unset).
  • ALL in-scope adapter groups migrated with *_reported seams and exhaustive no-wildcard *_error_from_report conversions; full tuple/transparent elimination (final sweep: zero error(transparent) left in mando-lib/src/adapter, archiver included).
  • Auth providers migrated first under the hard no-bodies/no-tokens rule (method/url/status only); sentinel leak tests everywhere.
  • PyO3 map_report_error: forward plumbing only; the wiring is grep-proven unreachable this phase.
  • Shared error_json_field_value hook unifying the error.errors/error.details JSON parse across dd_formatter, py-mando/src/log_formatter.rs, and py-mando-simulation/src/log_formatter.rs.
  • Env-gated report-backed mock error: MANDO_DEBUG_MOCK_ERROR, default off; replaces the dev-deploy branch’s hardcoded-JSON mock (supersedes the “mock still on the typed arm” warning in the dev prototype section, effective once phase 3 merges).

Key architectural finding: walled service towers (formal spec exception)

Only volue/ems carries Reports to the step boundary in production

The volue/ems service towers now use Error = Report<VolueEmsError>. The other SEVEN adapter groups are WALLED: their service towers fix flat error types, and the pub(crate) seam visibility makes cross-crate wiring structurally impossible, so their rich chains flatten at the legacy wrappers until a services-phase tower restructure. Every wall was independently verified genuine during review. The phase 3 spec records this as a formal exception; future phases inherit it as the starting map.

Update 2026-07-24: the *_reported seam MECHANICS are gone (commit f771805e): adapters expose ONE fn per operation returning MandoResult, and the flattening moved to the tower edges (~50 consumer edits). The walls themselves stand - flat-typed towers still flatten rich chains - but the dual-fn seam pattern no longer exists. See Error-authoring DX overhaul (2026-07-24).

Process notes

  • 3 of the 14 tasks needed fix waves, all dead-code deletions per review theme 2 (the “delete unused code” precedent): opl deleted dead order code; et3000 and position_manager dead variants deleted on review.
  • The final whole-branch review caught a Critical no task gate could see: the mando-simulator CONSUMER crate still had 2 tuple-syntax construction sites. Task gates compiled -p mando_lib only, never the consumer crate; workspace-wide clippy is the only gate that catches cross-crate fallout. Companion lesson recorded in mando-ci-lib-only-test-gate-2026-07-22.
  • The final fix wave also de-transparented 4 remaining variants after the reviewer ruled spec line 157 absolute: transparent named-source singles duplicate the inner level’s message in error.details.

Final gates at e0184bc3

  • Workspace cargo clippy --release --all-features: exit 0.
  • 574 lib tests passed / 0 failed (--lib = CI parity, see mando-ci-lib-only-test-gate-2026-07-22).
  • Transparent sweep: zero hits.
  • No Cargo.lock churn.
  • Diff vs develop: 61 files, +4390/-1482.
  • py-mando / py-mando-simulation pytest DEFERRED to CI (no local venv; Nexus is VPN-gated).

Follow-up ticket bundle (final reviewer’s triage)

  1. Auth-provider PRE-EXISTING info!/debug! body logs (a real leak surface, out of phase scope).
  2. ErrorAttr ids on non-success terminals only.
  3. py delegation tests CI-only.
  4. data_platform AWS env test panic-safety.
  5. opl clear_strategies missing portfolio attr.
  6. map_err closure-form drift.

Plus the standing items: the flight_end_to_end one-liner import fix (mando-ci-lib-only-test-gate-2026-07-22) and the mando-core standalone uuid/serde build fix.

Branch state

develop moved again after the rebase: 6fae64a5, the merge of feature/BE-2262-bess-am-poc (BESS AM (BE-2262) - mando-bess-am). The phase 3 branch is 1 merge behind; a pre-MR rebase is possible when the MR gets the go.

Update 2026-07-23 evening: resolved. The branch went to review as MR !585 at head a58df341 (22 commits, includes the short error.code commit) with no conflicts against develop and a green branch pipeline on the same sha - see MR !585 opened (2026-07-23 evening).

Short error.code decision (2026-07-23)

New decision by Andras (2026-07-23), amending field-contract decision 2 in the locked decisions above: error.code now carries the SHORT readable form, honoring the team spec’s line-100 mobile-readability intent: Enum::Variant (e.g. VolueEmsError::AuthenticationError), bare struct name for variantless types. error.kind, error.type, error.errors, error.details codes, and error.fingerprint ALL stay full module path.

Implementation (commit a58df341, the MR !585 head):

  • short_error_code resolver in mando-core; PascalCase-segment heuristic, valid because house naming keeps type/variant segments PascalCase and module segments snake_case.
  • Applied at the 4 macro emission sites ONLY; span/boundary layers inherit the short form as passthrough.
  • The short/full split is pinned by test kind_type_fingerprint_stay_full_while_code_goes_short; 577/0 workspace lib tests at the commit.
  • Verified LIVE in the local host-process run and forwarded to Datadog under env:dev-local-levander.

Consequences:

  1. The “mobile readability rationale is dead” pending team sentence is WITHDRAWN: conformance with the team spec is restored.
  2. The DD cutover impact for @error.code-keyed assets shrinks: the short form resembles the legacy codes (cutover callout below updated).
  3. @error.source is a RETIRED field: renamed to error.message in the redesign, pinned by a mando-core regression test. Any facet or monitor still keyed on it must be deleted or rekeyed onto @error.message / @error.errors. Andras hit exactly this today: an empty error.source consumer, since fixed by him.

Update 2026-07-24: extended one level deeper - error.details[].code also goes short (commit 4e519269); errors/kind/type/fingerprint stay full path. See Error-authoring DX overhaul (2026-07-24).

MR !585 opened (2026-07-23 evening)

Phase 3 went to review as MR !585 “feat: adopt error stack across all adapters”, description Closes BE-3657 (GitLab default per house ceremony), feature/BE-3657-phase3 → develop at head a58df341 (22 commits). No conflicts against develop; the branch pipeline is green on the same sha. Reviewers: krisztian.fekete1, gabor.nagy6, balint.budavoelgyi, jozsef.nagy1. The head commit is the short error.code implementation above, so it is in review scope.

Update 2026-07-24: the branch/MR head moved to 1f9cd611 “feat: render error stack as caused by chain” (23 commits, pushed + ls-remote verified) - the Caused-by error.stack renderer below rides the same review.

Update 2026-07-24 (later): the head moved again to f771805e “refactor: collapse adapter seams into native report signatures” (~27 commits, pushed) - the whole error-authoring DX overhaul (report vocabulary + short details codes + native signatures) rides the same review. See Error-authoring DX overhaul (2026-07-24).

Status 2026-07-24 (verified): MR !585 (head f771805e) is still OPEN with zero review comments; has_conflicts is false against develop even after develop took the BE-3685 merge daf6b6df; the head pipeline is success; the only unmet merge gate is not_approved.

error.stack Caused-by renderer decision (2026-07-24)

New decision by Andras (2026-07-24), REVISING locked decision 3: error.stack rendering switches from the error-stack crate’s Debug tree to a custom “Caused by” chain renderer. Trigger: Datadog collapses newlines in several views, and the crate’s ASCII connectors (|-, lone |, |->) turned into unreadable pipe soup in the real DD UI. The prefix-style renderer the original trade-off rejected is now adopted.

New format per level:

  • Head line {short code}: {message} - first level bare, every subsequent level prefixed Caused by: .
  • Location line at file:line:column.
  • One with key = value line per ErrorAttr.
  • Opaque attachments are no longer mentioned.

Implementation (commit 1f9cd611 on feature/BE-3657-phase3, the new MR !585 head):

  • render_stack_tree rewritten to walk frames the same way report_details does.
  • The crate debug-hook plumbing (install_debug_hook, Charset/ColorMode, HOOK_INIT) deleted as dead.
  • 579/0 workspace lib tests, including an exact multiline golden pin of the new format.
  • Verified LIVE locally and forwarded to Datadog: the 2026-07-24 events under env:dev-local-levander carry the new stacks AND the short error.code together.

Datadog research backing the decision

Findings from the Datadog docs, worth keeping:

  • Error Tracking for logs requires error-level status + service + (error.kind OR a valid error.stack). mando always emits error.kind, so the stack format never gates tracking/grouping for us.
  • DD’s “valid stack trace” bar is at least two lines with one meaningful frame carrying a FUNCTION NAME plus a filename. error_stack Locations carry no function names, so mando stacks can never be frame-parsed by DD regardless of format.
  • The source tag drives auto-parsing only for conventional language stacks; log remappers relocate attributes but do not parse.

Conclusion recorded: mando’s error.stack is purely presentational - optimize for human readability and graceful newline-collapse, which the Caused by: separators survive.

Error-authoring DX overhaul (2026-07-24)

Decision series by Andras (2026-07-24). North star: error reporting must not break Rust’s natural flow - ? is the API. Three commits landed on feature/BE-3657-phase3 (head now f771805e, ~27 commits, pushed, riding MR !585), and the next MR - a derive(MandoReport) proc macro - is designed and spec’d.

Report vocabulary (commit f5beabd4)

refactor: adopt report vocabulary for error authoring:

  • new_coded / CodedExt / change_context_coded DELETED workspace-wide.
  • Replaced by: ToReport::into_report (error-stack 0.8 already exports an IntoReport trait, hence the ToReport trait name; the METHOD name stays unambiguous), Result::reported(), .wrap(), and .attr(k, v) sugar.
  • flatten_report for Clone enums: 6 conversion fns became one-liners.
  • 307 sites swept across 37 files.

Bonus find: 20 degraded-location sites fixed for free

20 map_err(new_coded) FN-POINTER sites existed - the exact #[track_caller]-losing form - meaning those sites had degraded (non-call-site) locations until this commit swept them into the new vocabulary.

Short error.details codes (commit 4e519269)

feat: shorten error detail codes - extends the 2026-07-23 short error.code decision one level deeper:

  • error.details[].code now carries the SHORT form: the details panel is the human panel, and each entry’s file:line already carries the precision.
  • error.errors, error.kind, error.type, error.fingerprint stay full module path.
  • The short/full split is pinned by a test asserting both forms within ONE emission.

Native report signatures (commit f771805e)

refactor: collapse adapter seams into native report signatures - REVISES phase 3 locked decision 2. Andras’s verdict on the dual *_reported seam pattern: “not fit for a codebase this big”.

  • ~35 flat twins deleted; adapters now expose ONE fn per operation, returning MandoResult.
  • New bare newtype in mando-core: MandoReport<C>(Report<C>) with Deref, as_report, into_inner, delegating Display/Debug, a std::error::Error impl with source() = None, and a plain From<Report<C>> enabling ? auto-lift; plus a MandoResult<T, E> alias.
  • ~50 consumer edits: flattening moved to the tower edges.
  • The volue/ems towers keep Error = Report<VolueEmsError> internally, bridging with into_inner at 5 sites.
  • 5 dead conversion fns deleted on the branch’s own theme-2 precedent.
  • Gates: 580/0 workspace lib tests, clippy clean.

Next MR designed: derive(MandoReport) (spec’d 2026-07-24)

Spec: docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md (untracked, never commit). Goal: #[derive(MandoReport)] generating From impls so error handling becomes plain ? everywhere.

  • #[report(from(SrcType))]: display-mapping DEFAULT (preserves the Clone/message architecture).
  • #[report(wraps(Inner, message = "..."))]: chain levels.
  • MandoError marker trait bounds at boundaries + a clippy ban on Report::new.

Spike results (scratchpad/spike-trackcaller, rustc 1.89.0, error-stack 0.8.0):

  • Direct From impls on Report<LocalEnum> are ORPHAN-BLOCKED (E0117): the naive design is dead on arrival.
  • Through the local newtype, #[track_caller] propagates ?-site locations EXACTLY; without the attribute, locations degrade to the from body.
  • Andras had offered to sacrifice site tracking for the ergonomics; the spike proved no sacrifice is needed.

The spec grounding pass caught two design regressions BEFORE implementation:

  1. from(Src) must DISPLAY-MAP: carrying source fields would kill the Clone/flatten architecture.
  2. wraps() needs an explicit static message for struct variants.

Datadog operational gotchas (2026-07-24)

  • Facet queries with :: in the value MUST be quoted: @error.code:"VolueEmsError::ApiError". Unquoted, the query silently matches nothing.
  • Stale server held port 8081: the instance left running from the 2026-07-23 session survived and kept the port, so a new launch died with AddrInUse WHILE the old build’s old-format emissions kept flowing - a confusing mixed signal when iterating on log formats. Diagnose with lsof on the port and inspect the holder’s binary path.

Local live stream operational (2026-07-24)

  • Server on 8081 (build carrying the DX batch); a CONTINUOUS forwarder now ships error/warning lines to Datadog EU within ~5s under env:dev-local-levander.
  • MANDO_ENVIRONMENT=dev-local-levander is now set AT SOURCE, so the emitted ddtags are correct without the forward-time rewrite from the 2026-07-23 epilogue (the epilogue).
  • Complete field contract verified LIVE: error.code short, error.errors full, error.details codes short per level, error.fingerprint full, Caused-by error.stack, nested http.request/http.response blocks on 5xx paths.

Local end-to-end verification (2026-07-23)

Full LOCAL end-to-end test of the phase 3 error_stack telemetry. Every emission below came from the real runtime, not tests.

Setup: phase-3 branch binary (feature/BE-3657-phase3 @ b394e0d4, rebase-descendant of the reviewed e0184bc3, built in .worktrees/BE-3657-phase3) run as a HOST process against the mando-cli local stack (mando up --datadog -p infra: postgres + wiremock + local-dd-agent). Env assembled from .cargo/config.toml.example with overrides; env file kept at the session scratchpad (mando-p3.env). Full run recipe + friction log: mando-local-host-run-recipe-2026-07-23.

Verified emissions

  • Boot mock (MANDO_DEBUG_MOCK_ERROR=true): 4-level report; error.errors array of 4 full-path codes; error.details entries with real file:line AND attributes (data_group on the top level); error.stack box tree including the ErrorAttr sub-line and “1 additional opaque attachment” (the HttpContext); code == kind == type full path; root-cause error.message; trace correlation. (Both the box-tree stack shape and the opaque-attachment mention here are superseded 2026-07-24 - see error.stack Caused-by renderer decision (2026-07-24).)
  • Volue EMS spot step (404 from unstubbed wiremock): single-level coded emission with fingerprint code|step_path (data_update.load_battery_momentary_data), file:line spot_data.rs:79, static-first boundary message.
  • Volue EMS retrieve step: TWO-LEVEL chain in production: VolueEmsError::AuthenticationError + VolueEmsAuthenticationProviderError::FailedRequest; details entries at ems.rs:217 and ems_auth_provider.rs:155; chain-carrying boundary message Finished running step with errors: authentication error: '...': failed request (status: 404). This is the auth level the final-review fix wave added to the pilot retrieve path, confirmed live.
  • Metis merit order step: full-path error.code (MetisError::AuthenticationError) with NO error.errors array - the documented walled-tower flattening (Key architectural finding: walled service towers (formal spec exception)), the mixed world working as designed.
  • dd_formatter: renders error.errors/error.details as REAL nested JSON in the log lines (the shared error_json_field_value parse hook).

Datadog forwarding

12 error lines forwarded to Datadog EU via DIRECT INTAKE under env:dev-local-levander service:bess-os-service-mando. Host processes are not tailed by the local dd agent (its include filter scopes to mando-* containers, see mando-cli-datadog-local-logs-2026-07-22), so direct intake is the host-process path.

Session end state

Server left RUNNING at session end

localhost:8081, health 200, execution history queryable. Stack stops with mando down; the host process with pkill mando_bess.

Update 2026-07-24: this survived instance later held port 8081 and made a fresh launch die with AddrInUse while its old-format emissions kept flowing - see Datadog operational gotchas (2026-07-24).

DD_SERVICE env fix (commit ab8cfd11, 2026-07-22)

Rider commit on this branch replacing the hardcoded Datadog service name "bess-os-service-mando" in mando-lib with a DD_SERVICE env read, keeping the hardcoded value as fallback (also on empty string). It closes the mando-side root cause of the FR unsuffixed-service-name issue (fr-region-missing-datadog-logs-2026-07-21).

  • Implementation: resolve_dd_service() + a OnceLock-cached dd_service() accessor in mando-lib/src/app/mod.rs (app feature gated).
  • All 6 hardcode sites routed through it: the OTel Resource service name (get_tagging), the ddtags in dd_formatter.rs, and 4 raw global::meter("bess-os-service-mando") literals in service_base.rs (3x) and util/http_client_trace.rs that had bypassed the old const entirely.
  • Inline test covers default / empty-string / override behavior.
  • Deployment-neutral on the main stack: optimization-universe-iac terraform/bess_os_ecs.tf line 218 ALREADY sets DD_SERVICE = "bess-os-service-mando" on the mando container (algo containers and lambdas set their own names), so the code now honors what IaC says instead of ignoring it. The FR stack’s -fr-suffixed value starts winning once a build carrying this fix deploys there.

Dev prototype verification (build b7d15e2c)

Dev runs the throwaway build 1.16.1-feat.2682437891.b7d15e2c (branch feature/BE-3482-dev-deploy head b7d15e2c, static-first template merge; IaC pin applied 2026-07-16, Apply complete; NEVER merge, delete after the train lands). Stack = BE-3541 + BE-3656 + BE-3657 phases 1+2 + full-path codes + startup mock deep error + nested http.request/http.response + static-first boundary message + flow.step.system.

Verified in Datadog:

  • Nested error.details: 4-level chain with real file:line values.
  • error.errors array queries: element match, wildcard, and negation all work.
  • Nested http shape: http.{request{method,url,version,body,content_length}, response{status_code,body,content_length}}.
  • Trace correlation confirmed (closes the “verify dd.trace_id correlation” deferral); sample trace 0dc8d42739a08ac0805808b2d14b6644.

Mock still on the typed arm

The startup mock deep error fires once per task boot but still uses the typed error! arm with hardcoded JSON, NOT the report arm. Offer to upgrade it stands. Update 2026-07-23: superseded on the phase 3 branch - the mock is ported to mainline mando-bess as report-backed behind MANDO_DEBUG_MOCK_ERROR (default off). Effective once phase 3 merges; the throwaway dev-deploy branch keeps the old hardcoded-JSON form until then.

Pending DD cutover at BE-3657 merge

Datadog assets must be cut over when this deploys

All error.code / fingerprint values change format (short codes become full module paths) at the BE-3657 deploy. At merge time:

  1. Create facets: @error.fingerprint, @error.errors, @error.details.code.
  2. Inventory and update monitors keyed on SHORT codes, on flat http.* fields, and on the exact message "Failed running step".

Status 2026-07-23: !580 (phases 1+2) is MERGED, so this checklist becomes actionable with the first develop-based dev deploy carrying it.

Update 2026-07-23 evening: the short error.code decision (commit a58df341, in review with MR !585) shrinks the @error.code part of item 2: error.code stays Enum::Variant and closely resembles the legacy codes, so the format break narrows to error.kind/error.type, error.fingerprint, and the new array fields. NEW item 3: @error.source is RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message / @error.errors; one empty error.source consumer was found and fixed by Andras 2026-07-23.

Update 2026-07-24: error.details[].code also goes SHORT (commit 4e519269), so the @error.details.code facet keeps a readable short form; the full-path break remains on error.kind/error.type/error.fingerprint/error.errors. QUERY RULE for the new codes: facet values containing :: MUST be quoted (@error.code:"VolueEmsError::ApiError") - unquoted queries silently match nothing.

Relationship to prior direction

Supersedes the ErrorCode-derive / mando-lib-macro direction that only ever lived in worktree experiments (see Mando AGENTS.md Master Guide - there is NO ErrorCode derive on develop; error.kind/error.code are extracted at runtime by mando_core::error! from the Debug repr today). BE-3657 replaces that runtime Debug-repr extraction with static full-path codes captured at the error-construction site - originally via CodedExt, since 2026-07-24 via the ToReport report vocabulary (CodedExt is deleted, see Error-authoring DX overhaul (2026-07-24)).

Graph View

Backlinks