BE-3657 error_stack Adoption

Locked design decision to adopt the error_stack crate (=0.8.0) wholesale as mando’s error-handling substrate, replacing the ErrorCode-derive direction. Follow-on to BE-3656 APM Span Enrichment. Spec’d 2026-07-15/16; phases 1+2 MERGED 2026-07-23 via MR !580 (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 (plumbing + adapters) IN REVIEW since 2026-07-23 evening as MR !585 (feature/BE-3657-phase3 → develop, now @ f771805e, ~27 commits - see MR !585 opened (2026-07-23 evening)); review scope includes the 2026-07-23 short error.code decision (see Short error.code decision (2026-07-23)), the 2026-07-24 Caused-by error.stack renderer that REVISES locked decision 3 (see error.stack Caused-by renderer decision (2026-07-24)), and the 2026-07-24 error-authoring DX overhaul (report vocabulary, short error.details codes, native report signatures via the MandoReport newtype; the next MR, a derive(MandoReport) proc macro, is already spec’d - see Error-authoring DX overhaul (2026-07-24)). On 2026-07-27 both 2026-07-24 format experiments were reverted - the Caused-by stack renderer went back to error-stack’s default Debug tree (commit e813a84c) and the SHORT error.code/error.details[].code decisions were reversed to FULL module path (commit 4df2562a, short_error_code deleted) - the MandoReport name was finalized for the newtype AND the derive, and the derive(MandoReport) MR was started on the stacked branch feature/BE-3657-derive and, by session end, IMPLEMENTED IN FULL and pushed (574d757a, 16 commits, gates green; the MR itself deferred until !585 merges) - see Format decisions reverted, name locked, derive MR started (2026-07-27) and Derive MR complete and pushed (commit 574d757a, 2026-07-27). A follow-on five-lens assurance pass + fix wave (Andras-requested) then moved the derive tip to 86f3439f (19 commits): five parallel Opus lenses returned CONFORMANT / no-confirmed-bugs, and a 3-commit fix wave hardened the derive parser, added #[diagnostic::on_unimplemented] route-discovery diagnostics, and collapsed the metis auth-message duplication via the unwrap-after-bridge form - see Five-lens assurance pass + fix wave (2026-07-27, tip 86f3439f). MR !585 then took its review round 1 (krisztian.fekete1, 3 comments): the ebs upload_file restructure was confirmed an out-of-scope SMB mix-in and reverted (commit ee84ea50, now the MR !585 head) - see Review round 1 (2026-07-27, krisztian.fekete1). On 2026-07-28 Andras drove a SECOND DX overhaul of the derive branch (4 commits, tip 86f3439f → b7b04b71, 21 total): native MandoReport birth - into_report()/reported() return MandoReport/MandoResult at construction, the newtype’s std Error impl removed (the anyhow trick) so a blanket From is coherent and bare ? works for EVERY source, #[cause(Src, ...)] source lists plus a single #[mando_error] enum annotation replace the 07-27 bridge grammar and the #[derive(MandoReport)] export, and .mapped()/MappedExt are deleted; verified byte-identical (A/B + 3 location goldens) with wire e2e PASS vs 86f3439f - see DX overhaul v2: native MandoReport birth (2026-07-28). On 2026-07-28 (later) the derive branch was RENAMED to feature/BE-4000 (its own Jira key; worktree dir stays .worktrees/BE-3657-derive), a SECOND assurance pass (3 fresh Opus lenses over the v2 delta 86f3439f..cd5d53a7, because the 07-27 five-lens pass predated the DX v2 commits) returned CLEAN on all six dimensions, and a 2-commit standards fix-wave moved the derive tip to cd5d53a7; on the phase-3 side the opl create_orders dead code was RESTORED per review round 1 (phase-3 tip 0427c68d, all 3 MR !585 comments now resolved) - see BE-4000 rename, DX v2 assurance, opl restore (2026-07-28). On 2026-07-28 (later still) MR !585 was REBASED onto develop 29ed8e34 (which had advanced 22 commits, opening a merge conflict) to clear the conflict: phase-3 tip 0427c68d → c3519e73, force-with-lease pushed, all 30 commits preserved (range-diff 27=/3!), pipeline SUCCESS on the rebased head - but the new commits RESET krisztian.fekete1’s approval (GitLab drops approvals on any new commit), so !585 is not_approved again and needs re-approval to merge; the rebase also surfaced a LANDMINE (a big-branch rebase can break the build via NEW develop consumers of an API your branch migrated, invisible to textual-conflict detection, only caught by building) - see MR !585 rebased onto develop, pipeline green (2026-07-28). The dev environment runs a throwaway build carrying the phase 1+2 stack, field shapes verified in Datadog (see the dev prototype section).

For Agents

Status 2026-07-24: MR !580 (phases 1+2) MERGED (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 IN REVIEW as MR !585 “feat: adopt error stack across all adapters” (feature/BE-3657-phase3 → develop, head f771805e, ~27 commits, pushed; reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1; branch pipeline was green at a58df341, local gates at f771805e: 580/0 workspace lib tests + clippy clean) - see MR !585 opened (2026-07-23 evening). In review scope: the 2026-07-23 SHORT error.code decision (commit a58df341: Enum::Variant on error.code only) - see Short error.code decision (2026-07-23) - the 2026-07-24 Caused-by error.stack renderer (commit 1f9cd611: custom “Caused by” chain renderer replaces the crate’s Debug tree, REVISES locked decision 3) - see error.stack Caused-by renderer decision (2026-07-24) - and the 2026-07-24 error-authoring DX OVERHAUL (commits f5beabd4 report vocabulary / 4e519269 short error.details codes / f771805e native report signatures + MandoReport newtype) - see Error-authoring DX overhaul (2026-07-24). Execution history: Phase 3 executed (2026-07-22 evening through 2026-07-23). Worktree .worktrees/BE-3657-phase3. Specs: docs/superpowers/specs/2026-07-15-error-stack-adoption-design.md (phases 1+2, includes the flipped template + annotation direction) + docs/superpowers/specs/2026-07-22-be3657-phase3-adapters-design.md (phase 3; now records the walled-towers formal exception) + docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md (next MR: derive(MandoReport) From-impl generation). Plans: docs/superpowers/plans/2026-07-15-be3657-error-stack-phase1.md + 2026-07-16-be3657-error-stack-phase2.md + 2026-07-22-be3657-error-stack-phase3.md. All untracked, never commit. Remaining: the spec’d derive(MandoReport) MR (next up), phase 4 (#[error_meta(...)] annotations, Andras’s own work) + the follow-up ticket bundle in the execution section. Phase 3 telemetry VERIFIED LOCALLY end to end 2026-07-23 from the real runtime (binary b394e0d4, a rebase-descendant; the short-code commit a58df341 and the Caused-by-renderer commit 1f9cd611 also verified live) and the DX batch is verified live 2026-07-24 through a CONTINUOUS forwarder to Datadog (env:dev-local-levander, ~5s latency; complete field contract confirmed: code short, errors full, details codes short per level, fingerprint full, Caused-by stack, nested http on 5xx paths) - see Local end-to-end verification (2026-07-23) + Local live stream operational (2026-07-24); host-process run recipe: mando-local-host-run-recipe-2026-07-23. Status 2026-07-27: MR !585 head moved to 4df2562a, then ee84ea50 (review round 1). BOTH 2026-07-24 format experiments were reverted: the Caused-by error.stack renderer went back to the crate’s default Debug tree (commit e813a84c, RE-REVISES locked decision 3 toward its original direction), and the SHORT error.code/error.details[].code decisions were REVERSED to FULL module path (commit 4df2562a) - every code-bearing field is uniform full path again, short_error_code helper deleted, 575/0 lib tests. MandoReport is the FINAL name for the newtype AND the derive; the derive(MandoReport) MR is IN PROGRESS stacked off the phase-3 tip (branch feature/BE-3657-derive, fast-forwarded to ee84ea50). MR !585’s review round 1 landed 2026-07-27 (krisztian.fekete1, 3 comments): the ebs upload_file restructure was confirmed an out-of-scope SMB mix-in and reverted (commit ee84ea50, the new head); the MANDO_DEBUG_MOCK_ERROR mock gate and the opl create_orders dead-code deletion were questions answered in-thread. Still OPEN, not_approved the only gate - see Format decisions reverted, name locked, derive MR started (2026-07-27) and Review round 1 (2026-07-27, krisztian.fekete1). The derive(MandoReport) EXECUTION then began on feature/BE-3657-derive (Tasks 1-3 landed + review-approved: 0a083ecf / c8c0370c / edf0dd32) and hit an ORPHAN-RULE flaw the 2026-07-24 single-crate spike had masked; Andras resolved it with the bridge design (mando-core MandoFrom / MandoWraps local traits + MappedExt / WrappedExt ext traits; grammar split from = local, NEW mapped = any source, wraps = chain) - see Derive execution and the bridge-design pivot (2026-07-27). The derive is now IMPLEMENTATION-COMPLETE and pushed (feature/BE-3657-derive @ 574d757a, 16 commits on base ee84ea50, ls-remote verified; the full 14-task plan + a 6-item fix wave, per-task adversarial review + a whole-branch final review on Fable that returned “with fixes” then re-approved with zero new issues; clippy clean, 586/0/32 lib tests). Shipped: the MandoError marker + wrap_report + the bridge traits, the #[derive(MandoReport)] proc macro (bare/from/mapped/wraps grammars, frozen derive-time strings, 12-case trybuild suite), enforcement (MandoError bounds on the 4 boundary sites + a clippy Report::new ban), and six adapter sweep waves (~50 call-site collapses + a shared Utf8BodyError alias). Second execution discovery = the TOWER FREEZE law (sites inside raw-Report-typed tower fns cannot collapse; auth-wrap score 2 live vs 5 frozen). Verified byte-identical to the verbose forms (A/B suite + track_caller goldens + e2e wire compare). A follow-on Andras-requested five-lens assurance pass then hardened the branch to tip 86f3439f (19 commits): five parallel Opus lenses returned CONFORMANT / no-confirmed-bugs, and a 3-commit fix wave added derive-parser hardening (empty-attr derive-time errors, UI suite 12 → 15), #[diagnostic::on_unimplemented] route-discovery diagnostics, and a metis DRY fix via the unwrap-after-bridge form (proven able to unfreeze all 5 walled auth towers, deliberately not applied beyond metis) - see Five-lens assurance pass + fix wave (2026-07-27, tip 86f3439f). DX v2 (2026-07-28, tip b7b04b71, 4 commits/21 total): a SECOND Andras-driven DX overhaul reached bare ? for EVERY source + a single #[mando_error] enum annotation. Native MandoReport birth: into_report()/reported() return MandoReport/MandoResult at CONSTRUCTION, the newtype’s std Error impl REMOVED (the anyhow trick) so the blanket From<S: MandoFrom> is coherent (5acd801b); #[cause(Src1, Src2)] bare source lists replace the 07-27 from/mapped bridge grammar with .mapped()/MappedExt DELETED and wraps/.wrapped()? kept (8ebd6ad9); the #[mando_error] attribute macro (Error + Debug + marker + routes in one annotation, shared generate_routes engine) replaces #[derive(MandoReport)], whose export is DELETED (4221f00b + a 28-enum sweep b7b04b71). SECOND scratch-crate false-green (coherence this time, orphan on 07-27 - scratch crates lie about the open world); the .lift() measurement found 227 seams not ~11, ~160 vanishing under native birth (towers pay .into_inner() at ~40-50 sites). Byte-identical (A/B + 3 goldens) + wire e2e PASS vs 86f3439f; intended delta: mando-simulator init moved to the report arm - see DX overhaul v2: native MandoReport birth (2026-07-28). The MR itself waits for !585 to merge, then git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000 (the derive branch was RENAMED from feature/BE-3657-derive to feature/BE-4000 on 2026-07-28) + re-gate + Andras’s explicit yes - see Derive MR complete and pushed (commit 574d757a, 2026-07-27) and BE-4000 rename, DX v2 assurance, opl restore (2026-07-28). Current tips (2026-07-28): BE-4000 @ cd5d53a7 (DX v2 double-assured); MR !585 phase 3 @ c3519e73 (REBASED onto develop 29ed8e34, pipeline green, not_approved - the rebase reset krisztian’s approval); develop @ 29ed8e34. See MR !585 rebased onto develop, pipeline green (2026-07-28).

Locked decisions (2026-07-15/16, Andras)

1. Full error_stack (=0.8.0) adoption over an ErrorCode-derive macro. Guiding principle: “leave what we can to error_stack” - mando contributes ONLY:
   • the ErrorCode attachment (full-path code captured statically in a CodedExt helper at each change_context / new site),
   • extraction functions, and
   • an error!(report = ...) macro arm. Everything else (chaining, source walking, rendering) is the crate’s job. Amended 2026-07-24: the authoring vocabulary changed - CodedExt / new_coded / change_context_coded are DELETED workspace-wide, replaced by ToReport::into_report / Result::reported() / .wrap() / .attr(k, v); the principle itself stands - see Error-authoring DX overhaul (2026-07-24).
2. Field contract (single outermost identity across all three):
   • error.code == error.kind == error.type == the outermost full-path code.
   • error.fingerprint uses the full-path code.
   • error.errors = array of codes.
   • error.details = array of {code, file, line, message} objects.
   • Amended 2026-07-23: error.code alone now carries the SHORT readable form (Enum::Variant; bare struct name for variantless types); kind/type/errors/details codes/fingerprint stay full module path - see Short error.code decision (2026-07-23).
   • Amended again 2026-07-24: error.details[].code joins the short form (the details panel is the human panel; each entry’s file:line carries the precision); kind/type/errors/fingerprint stay full path - see Error-authoring DX overhaul (2026-07-24).
   • REVERSED 2026-07-27: both short-code amendments above are undone (commit 4df2562a) - error.code and error.details[].code are FULL module path again, so decision 2’s ORIGINAL “single outermost identity, full path everywhere” holds uniformly across all code-bearing fields - see Short codes reverted to full path (commit 4df2562a).
3. Tree rendering = the crate’s own output, with codes attached as sub-lines (attachment style), NOT a prefix style. Accepted trade-off. REVISED 2026-07-24: after seeing the tree in the real Datadog UI, the crate’s Debug tree is dropped for a custom prefix-style “Caused by” chain renderer - see error.stack Caused-by renderer decision (2026-07-24). RE-REVERTED 2026-07-27: back to the crate’s default Debug tree (ErrorCode debug hook, ASCII charset, no color, full-path codes per frame) - the 2026-07-24 custom renderer is gone and this decision returns to its ORIGINAL direction; the Datadog newline-collapse quirk is accepted as display-only - see error.stack reverted to the crate default tree (commit e813a84c).
4. Boundary message template FLIPPED to STATIC-FIRST on 2026-07-16: "{static}: {outermost}[: {chain}]". Supersedes the earlier outermost-first order ("{outermost}: {static}: {chain}") recorded on 07-15/16; the flip is in the authoritative spec and deployed in the dev prototype build.
5. Phase-2+ annotation direction (Andras’s own work once the phase-1 plumbing exists, must not be precluded by phase 1): per-variant #[error_meta(event_type, resolution step, business_message, description)] feeding error.details enrichment plus resolution-doc codegen. error_stack attachments are open-ended, so phase 1 does not block this.
6. Bodies never on spans (shared with BE-3656): request/response bodies attach only via error events, a deliberate deviation from the logs-and-apm.md spec table, STILL to be raised with the team.
7. error.errors / error.details excluded from span tags: the arrays live on the log side only (Datadog flattens array span attributes to key.0/key.1, breaking facets); spans keep the scalar error.code.
8. First-error-wins span enrichment guard (from BE-3656): a marker extension prevents later errors overwriting span error tags, since set_attribute appends under the 128-attr SDK cap.
9. HTTP fields nested as http.request.* / http.response.* (axum semconv-aligned; collision investigated, none).

Build state (2026-07-22)

Phases 1+2 are done and pushed. Morning state:

• 13 commits @ 2baea68a atop the BE-3656 head 8f50ef2a (feature/BE-3657 on origin, ls-remote verified). Contains BOTH phases; suggested MR reflects that.
• Rebased 2026-07-21. During the rebase, 3 develop-inherited bare-code fallback tests were aligned to full-path expectations and squashed into the commit “feat: use full module path in error code”.
• Gates green via the workspace build graph (mando_core 94/0 through the workspace). Standalone cargo build -p mando_core fails on a PRE-EXISTING uuid/serde compile issue inherited from the BE-3643 merge (bf953ec8), not caused by this branch.
• Later phases remain: phase 3 (planned 2026-07-22 evening as plumbing + adapters, superseding the earlier “workspace breadth” framing - see Phase 3 planned (2026-07-22 evening); the py-mando formatter parse-hook port and the tuple/transparent restructuring are IN it) and phase 4 (the #[error_meta(...)] annotations, Andras’s own work).

Afternoon 2026-07-22: rebased onto develop after !578 merged

• MR !578 merged (squash 46a9be1b, merge 4a0d297f = new develop tip), so the pending rebase --onto develop happened: all 13 commits replayed zero-conflict, head 2baea68a became 22ccae93, patch-id verified byte-identical to the reviewed content.
• Branch CI pipeline 2696324182 on 22ccae93: success.
• Then commit ab8cfd11 “fix: read dd service name from env” was appended (Andras asked to sneak it into the upcoming MR), making the branch 14 commits @ ab8cfd11, pushed and ls-remote verified. Details in DD_SERVICE env fix (commit ab8cfd11, 2026-07-22).
• Local gates green after the append: clippy clean, 516 tests passed / 0 failed at lib scope (--lib = CI parity, see mando-ci-lib-only-test-gate-2026-07-22).
• MR ceremony: nothing left but Andras’s explicit yes.

Phase 3 planned (2026-07-22 evening)

Phase 3 design approved by Andras 2026-07-22 (evening session). Kept as the planning record; execution ran the same evening through 2026-07-23, see Phase 3 executed (2026-07-22 evening through 2026-07-23).

Artifacts (untracked, never commit): spec docs/superpowers/specs/2026-07-22-be3657-phase3-adapters-design.md; plan docs/superpowers/plans/2026-07-22-be3657-error-stack-phase3.md (15 tasks).

Locked phase 3 decisions

1. Scope = “plumbing + adapters”. Rejected alternatives: plumbing-only; full workspace breadth.
2. Mechanics = seam replication per the Volue pilot. Rejected alternatives: native Report signatures; boundary-only wrapping. Pilot artifacts already on the base branch: retrieve_ts_data_reported, send_request_reported, volue_error_from_report in adapter/volue/ems/ts_data_retrieve.rs. REVISED 2026-07-24: the seam pattern was eliminated on the same branch in favor of the once-rejected native report signatures, made viable by the MandoReport newtype’s ? auto-lift (commit f771805e) - see Error-authoring DX overhaul (2026-07-24).
3. Ticket = keep riding BE-3657: branch feature/BE-3657-phase3 off develop AFTER !580 merges; one phase = one MR.

Conformance pass against the team spec

The design was tightened against Balazs’s team spec (architecture-design repo, doc/content/doc/ops/logs-and-apm.md, fetched from main 2026-07-22):

1. Line 157 is absolute: ALL tuple/transparent variants in migrated enums get restructured.
2. error.details gains the spec’s attributes field (lines 150-155) via a new mando-core ErrorAttr attachment + extraction; this lands BEFORE any adapter migrates.
3. Line 159 http roll-up: implemented as an outermost-wins pick of the report’s HttpContext attachment; with_report fills http_context when unset. The spec’s “force with a macro” wording is interpreted positionally and goes on the pending team-sentences list.
4. Auth provider migrations carry a hard sensitive-data rule (lines 121-124): no bodies, no tokens; method/url/status only.

Adapter inventory and order (grounded on the branch)

1. Auth providers FIRST: alpiq/authentication + the Volue EMS/ATP auth providers (sensitive-data rule above applies).
2. volue/ems completion: spot + send join the ts_data_retrieve pilot.
3. volue/atp.
4. alpiq/ebs: SMB, so NO http roll-up; share/path ErrorAttr instead.
5. alpiq mdr / opl / data_platform / et_3000.
6. fingrid.
7. alpiq/metis.
8. mando group: rest / algo / simulator / ms_teams / microsoft.
9. alpiq/position_manager.

Flagged defaults (Andras can veto before execution)

Two vetoable defaults baked into the plan

1. adapter/mando/archiver (+ its postgres submodule) DEFERRED to the repos phase: data-plane, separate CI harness, no step boundary.
2. The startup mock deep error gets PORTED to mainline mando-bess (it exists only on the throwaway dev-deploy branch) behind MANDO_DEBUG_MOCK_ERROR, default-off; one IaC dev env line at deploy time; droppable if the team objects.

New mando-core surface (phase 3 task 1)

• pub struct ErrorAttr(pub String, pub serde_json::Value) attachment.
• error_attr_value: one-level-deeper rendering for error-typed attributes.
• report_first_attachment: outermost-wins generic pick.
• report_details entries gain attributes.
• A shared error_json_field_value fn unifies the error.errors/error.details JSON parsing across dd_formatter and both python package log formatters (py-mando/src/log_formatter.rs, py-mando-simulation/src/log_formatter.rs).

Phase 3 executed (2026-07-22 evening through 2026-07-23)

Phase 3 ran subagent-driven on branch feature/BE-3657-phase3: 14 tasks + a final whole-branch review, every task under an adversarial spec + quality review. The branch was initially stacked on the in-review !580 head ab8cfd11; after !580 merged, rebase --onto develop replayed with zero conflicts, patch-id byte-identical. 20 commits at execution end, review HEAD e0184bc3, PUSHED (ls-remote verified). Update 2026-07-23 evening: 22 commits @ a58df341, IN REVIEW as MR !585 - see MR !585 opened (2026-07-23 evening). Neither flagged default was vetoed: the archiver seam migration stayed deferred per the plan (the transparent sweep still covered adapter/mando/archiver), and the mock port shipped env-gated.

Delivered

• mando-core surface: ErrorAttr attachment + per-level attributes on error.details entries; error_attr_value one-level nesting for error-typed attribute values; report_first_attachment outermost-wins generic pick; with_report http roll-up (fills http_context from the report’s outermost HttpContext when unset).
• ALL in-scope adapter groups migrated with *_reported seams and exhaustive no-wildcard *_error_from_report conversions; full tuple/transparent elimination (final sweep: zero error(transparent) left in mando-lib/src/adapter, archiver included).
• Auth providers migrated first under the hard no-bodies/no-tokens rule (method/url/status only); sentinel leak tests everywhere.
• PyO3 map_report_error: forward plumbing only; the wiring is grep-proven unreachable this phase.
• Shared error_json_field_value hook unifying the error.errors/error.details JSON parse across dd_formatter, py-mando/src/log_formatter.rs, and py-mando-simulation/src/log_formatter.rs.
• Env-gated report-backed mock error: MANDO_DEBUG_MOCK_ERROR, default off; replaces the dev-deploy branch’s hardcoded-JSON mock (supersedes the “mock still on the typed arm” warning in the dev prototype section, effective once phase 3 merges).

Key architectural finding: walled service towers (formal spec exception)

Only volue/ems carries Reports to the step boundary in production

The volue/ems service towers now use Error = Report<VolueEmsError>. The other SEVEN adapter groups are WALLED: their service towers fix flat error types, and the pub(crate) seam visibility makes cross-crate wiring structurally impossible, so their rich chains flatten at the legacy wrappers until a services-phase tower restructure. Every wall was independently verified genuine during review. The phase 3 spec records this as a formal exception; future phases inherit it as the starting map.

Update 2026-07-24: the *_reported seam MECHANICS are gone (commit f771805e): adapters expose ONE fn per operation returning MandoResult, and the flattening moved to the tower edges (~50 consumer edits). The walls themselves stand - flat-typed towers still flatten rich chains - but the dual-fn seam pattern no longer exists. See Error-authoring DX overhaul (2026-07-24).

Process notes

• 3 of the 14 tasks needed fix waves, all dead-code deletions per review theme 2 (the “delete unused code” precedent): opl deleted dead order code; et3000 and position_manager dead variants deleted on review.
• The final whole-branch review caught a Critical no task gate could see: the mando-simulator CONSUMER crate still had 2 tuple-syntax construction sites. Task gates compiled -p mando_lib only, never the consumer crate; workspace-wide clippy is the only gate that catches cross-crate fallout. Companion lesson recorded in mando-ci-lib-only-test-gate-2026-07-22.
• The final fix wave also de-transparented 4 remaining variants after the reviewer ruled spec line 157 absolute: transparent named-source singles duplicate the inner level’s message in error.details.

Final gates at e0184bc3

• Workspace cargo clippy --release --all-features: exit 0.
• 574 lib tests passed / 0 failed (--lib = CI parity, see mando-ci-lib-only-test-gate-2026-07-22).
• Transparent sweep: zero hits.
• No Cargo.lock churn.
• Diff vs develop: 61 files, +4390/-1482.
• py-mando / py-mando-simulation pytest DEFERRED to CI (no local venv; Nexus is VPN-gated).

Follow-up ticket bundle (final reviewer’s triage)

1. Auth-provider PRE-EXISTING info!/debug! body logs (a real leak surface, out of phase scope).
2. ErrorAttr ids on non-success terminals only.
3. py delegation tests CI-only.
4. data_platform AWS env test panic-safety.
5. opl clear_strategies missing portfolio attr.
6. map_err closure-form drift.

Plus the standing items: the flight_end_to_end one-liner import fix (mando-ci-lib-only-test-gate-2026-07-22) and the mando-core standalone uuid/serde build fix.

Branch state

develop moved again after the rebase: 6fae64a5, the merge of feature/BE-2262-bess-am-poc (BESS AM (BE-2262) - mando-bess-am). The phase 3 branch is 1 merge behind; a pre-MR rebase is possible when the MR gets the go.

Update 2026-07-23 evening: resolved. The branch went to review as MR !585 at head a58df341 (22 commits, includes the short error.code commit) with no conflicts against develop and a green branch pipeline on the same sha - see MR !585 opened (2026-07-23 evening).

Short error.code decision (2026-07-23)

Reversed 2026-07-27

This short error.code decision was REVERSED on 2026-07-27 (commit 4df2562a): error.code is FULL module path again and the short_error_code helper is deleted. The section below is retained for the trail - see Short codes reverted to full path (commit 4df2562a).

New decision by Andras (2026-07-23), amending field-contract decision 2 in the locked decisions above: error.code now carries the SHORT readable form, honoring the team spec’s line-100 mobile-readability intent: Enum::Variant (e.g. VolueEmsError::AuthenticationError), bare struct name for variantless types. error.kind, error.type, error.errors, error.details codes, and error.fingerprint ALL stay full module path.

Implementation (commit a58df341, the MR !585 head):

• short_error_code resolver in mando-core; PascalCase-segment heuristic, valid because house naming keeps type/variant segments PascalCase and module segments snake_case.
• Applied at the 4 macro emission sites ONLY; span/boundary layers inherit the short form as passthrough.
• The short/full split is pinned by test kind_type_fingerprint_stay_full_while_code_goes_short; 577/0 workspace lib tests at the commit.
• Verified LIVE in the local host-process run and forwarded to Datadog under env:dev-local-levander.

Consequences:

1. The “mobile readability rationale is dead” pending team sentence is WITHDRAWN: conformance with the team spec is restored.
2. The DD cutover impact for @error.code-keyed assets shrinks: the short form resembles the legacy codes (cutover callout below updated).
3. @error.source is a RETIRED field: renamed to error.message in the redesign, pinned by a mando-core regression test. Any facet or monitor still keyed on it must be deleted or rekeyed onto @error.message / @error.errors. Andras hit exactly this today: an empty error.source consumer, since fixed by him.

Update 2026-07-24: extended one level deeper - error.details[].code also goes short (commit 4e519269); errors/kind/type/fingerprint stay full path. See Error-authoring DX overhaul (2026-07-24).

MR !585 opened (2026-07-23 evening)

Phase 3 went to review as MR !585 “feat: adopt error stack across all adapters”, description Closes BE-3657 (GitLab default per house ceremony), feature/BE-3657-phase3 → develop at head a58df341 (22 commits). No conflicts against develop; the branch pipeline is green on the same sha. Reviewers: krisztian.fekete1, gabor.nagy6, balint.budavoelgyi, jozsef.nagy1. The head commit is the short error.code implementation above, so it is in review scope.

Update 2026-07-24: the branch/MR head moved to 1f9cd611 “feat: render error stack as caused by chain” (23 commits, pushed + ls-remote verified) - the Caused-by error.stack renderer below rides the same review.

Update 2026-07-24 (later): the head moved again to f771805e “refactor: collapse adapter seams into native report signatures” (~27 commits, pushed) - the whole error-authoring DX overhaul (report vocabulary + short details codes + native signatures) rides the same review. See Error-authoring DX overhaul (2026-07-24).

Status 2026-07-24 (verified): MR !585 (head f771805e) is still OPEN with zero review comments; has_conflicts is false against develop even after develop took the BE-3685 merge daf6b6df; the head pipeline is success; the only unmet merge gate is not_approved.

Update 2026-07-27: head moved to e813a84c (the error.stack revert - see error.stack reverted to the crate default tree (commit e813a84c)), then to 4df2562a (the short-code reversal - see Short codes reverted to full path (commit 4df2562a)), then to ee84ea50 (the ebs upload_file restore from review round 1 - see Review round 1 (2026-07-27, krisztian.fekete1)). Review round 1 arrived 2026-07-27 (krisztian.fekete1, 3 comments, the first on this MR); MR still OPEN, not_approved still the only merge gate. The derive(MandoReport) MR was cut off this tip (branch feature/BE-3657-derive) rather than waiting for the merge - see derive(MandoReport) MR started, stacked off phase3.

error.stack Caused-by renderer decision (2026-07-24)

Reverted 2026-07-27

This custom renderer was REVERTED to error-stack’s default Debug tree on 2026-07-27 (commit e813a84c). The decision below is retained for the trail; current behavior is the crate default tree - see error.stack reverted to the crate default tree (commit e813a84c).

New decision by Andras (2026-07-24), REVISING locked decision 3: error.stack rendering switches from the error-stack crate’s Debug tree to a custom “Caused by” chain renderer. Trigger: Datadog collapses newlines in several views, and the crate’s ASCII connectors (|-, lone |, |->) turned into unreadable pipe soup in the real DD UI. The prefix-style renderer the original trade-off rejected is now adopted.

New format per level:

• Head line {short code}: {message} - first level bare, every subsequent level prefixed Caused by: .
• Location line at file:line:column.
• One with key = value line per ErrorAttr.
• Opaque attachments are no longer mentioned.

Implementation (commit 1f9cd611 on feature/BE-3657-phase3, the new MR !585 head):

• render_stack_tree rewritten to walk frames the same way report_details does.
• The crate debug-hook plumbing (install_debug_hook, Charset/ColorMode, HOOK_INIT) deleted as dead.
• 579/0 workspace lib tests, including an exact multiline golden pin of the new format.
• Verified LIVE locally and forwarded to Datadog: the 2026-07-24 events under env:dev-local-levander carry the new stacks AND the short error.code together.

Datadog research backing the decision

Findings from the Datadog docs, worth keeping:

• Error Tracking for logs requires error-level status + service + (error.kind OR a valid error.stack). mando always emits error.kind, so the stack format never gates tracking/grouping for us.
• DD’s “valid stack trace” bar is at least two lines with one meaningful frame carrying a FUNCTION NAME plus a filename. error_stack Locations carry no function names, so mando stacks can never be frame-parsed by DD regardless of format.
• The source tag drives auto-parsing only for conventional language stacks; log remappers relocate attributes but do not parse.

Conclusion recorded: mando’s error.stack is purely presentational - optimize for human readability and graceful newline-collapse, which the Caused by: separators survive.

Error-authoring DX overhaul (2026-07-24)

Decision series by Andras (2026-07-24). North star: error reporting must not break Rust’s natural flow - ? is the API. Three commits landed on feature/BE-3657-phase3 (head now f771805e, ~27 commits, pushed, riding MR !585), and the next MR - a derive(MandoReport) proc macro - is designed and spec’d.

Report vocabulary (commit f5beabd4)

refactor: adopt report vocabulary for error authoring:

• new_coded / CodedExt / change_context_coded DELETED workspace-wide.
• Replaced by: ToReport::into_report (error-stack 0.8 already exports an IntoReport trait, hence the ToReport trait name; the METHOD name stays unambiguous), Result::reported(), .wrap(), and .attr(k, v) sugar.
• flatten_report for Clone enums: 6 conversion fns became one-liners.
• 307 sites swept across 37 files.

Bonus find: 20 degraded-location sites fixed for free

20 map_err(new_coded) FN-POINTER sites existed - the exact #[track_caller]-losing form - meaning those sites had degraded (non-call-site) locations until this commit swept them into the new vocabulary.

Short error.details codes (commit 4e519269)

Reversed 2026-07-27

Reverted together with the short error.code decision on 2026-07-27 (commit 4df2562a): error.details[].code is FULL module path again. Retained for the trail - see Short codes reverted to full path (commit 4df2562a).

feat: shorten error detail codes - extends the 2026-07-23 short error.code decision one level deeper:

• error.details[].code now carries the SHORT form: the details panel is the human panel, and each entry’s file:line already carries the precision.
• error.errors, error.kind, error.type, error.fingerprint stay full module path.
• The short/full split is pinned by a test asserting both forms within ONE emission.

Native report signatures (commit f771805e)

refactor: collapse adapter seams into native report signatures - REVISES phase 3 locked decision 2. Andras’s verdict on the dual *_reported seam pattern: “not fit for a codebase this big”.

• ~35 flat twins deleted; adapters now expose ONE fn per operation, returning MandoResult.
• New bare newtype in mando-core: MandoReport<C>(Report<C>) with Deref, as_report, into_inner, delegating Display/Debug, a std::error::Error impl with source() = None, and a plain From<Report<C>> enabling ? auto-lift; plus a MandoResult<T, E> alias.
• ~50 consumer edits: flattening moved to the tower edges.
• The volue/ems towers keep Error = Report<VolueEmsError> internally, bridging with into_inner at 5 sites.
• 5 dead conversion fns deleted on the branch’s own theme-2 precedent.
• Gates: 580/0 workspace lib tests, clippy clean.

Update 2026-07-28 (DX v2): into_report() / reported() now return MandoReport / MandoResult NATIVELY (the type is born mando-typed at construction, not lifted from Report at seams), and the newtype’s std::error::Error impl is REMOVED (the anyhow trick) so a blanket From<S: MandoFrom> becomes coherent - see DX overhaul v2: native MandoReport birth (2026-07-28).

Next MR designed: derive(MandoReport) (spec’d 2026-07-24)

Spec: docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md (untracked, never commit). Goal: #[derive(MandoReport)] generating From impls so error handling becomes plain ? everywhere.

• #[report(from(SrcType))]: display-mapping DEFAULT (preserves the Clone/message architecture).
• #[report(wraps(Inner, message = "..."))]: chain levels.
• MandoError marker trait bounds at boundaries + a clippy ban on Report::new.

Spike results (scratchpad/spike-trackcaller, rustc 1.89.0, error-stack 0.8.0):

• Direct From impls on Report<LocalEnum> are ORPHAN-BLOCKED (E0117): the naive design is dead on arrival.
• Through the local newtype, #[track_caller] propagates ?-site locations EXACTLY; without the attribute, locations degrade to the from body.
• Andras had offered to sacrifice site tracking for the ergonomics; the spike proved no sacrifice is needed.

The spec grounding pass caught two design regressions BEFORE implementation:

1. from(Src) must DISPLAY-MAP: carrying source fields would kill the Clone/flatten architecture.
2. wraps() needs an explicit static message for struct variants.

Update 2026-07-27: the name is FINALIZED as MandoReport (newtype + derive) and this MR is now IN PROGRESS on the stacked branch feature/BE-3657-derive - see Format decisions reverted, name locked, derive MR started (2026-07-27). During execution the two-arm grammar above (from + wraps) SPLIT into three: the orphan rule forced from() to crate-local sources only and added a mapped() arm for any source, both routed through new mando-core bridge traits - see Derive execution and the bridge-design pivot (2026-07-27).

Datadog operational gotchas (2026-07-24)

• Facet queries with :: in the value MUST be quoted: @error.code:"VolueEmsError::ApiError". Unquoted, the query silently matches nothing.
• Stale server held port 8081: the instance left running from the 2026-07-23 session survived and kept the port, so a new launch died with AddrInUse WHILE the old build’s old-format emissions kept flowing - a confusing mixed signal when iterating on log formats. Diagnose with lsof on the port and inspect the holder’s binary path.

Local live stream operational (2026-07-24)

• Server on 8081 (build carrying the DX batch); a CONTINUOUS forwarder now ships error/warning lines to Datadog EU within ~5s under env:dev-local-levander.
• MANDO_ENVIRONMENT=dev-local-levander is now set AT SOURCE, so the emitted ddtags are correct without the forward-time rewrite from the 2026-07-23 epilogue (the epilogue).
• Complete field contract verified LIVE: error.code short, error.errors full, error.details codes short per level, error.fingerprint full, Caused-by error.stack, nested http.request/http.response blocks on 5xx paths.

Format decisions reverted, name locked, derive MR started (2026-07-27)

The 2026-07-27 moves by Andras (session “Error hdl”), all on feature/BE-3657-phase3 / MR !585 or stacked off it. The through-line: BOTH 2026-07-24 format experiments (custom Caused-by stack renderer, short codes) were reverted, restoring the crate’s default stack tree AND uniform full-path codes across every code-bearing field. MR !585 head is now 4df2562a. The repo-side docs carry the detail (handover docs/superpowers/HANDOVER-error-handling-2026-07-24.md, spec docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, derive plan docs/superpowers/plans/2026-07-27-be3657-report-derive.md, all untracked); this is the vault decision trail.

error.stack reverted to the crate default tree (commit e813a84c)

RE-REVISES the error.stack Caused-by renderer decision (2026-07-24) (which had itself revised locked decision 3): the custom prefix-style “Caused by” chain renderer (commit 1f9cd611) is REVERTED back to error-stack’s DEFAULT Debug tree rendering - effectively a return to the ORIGINAL locked-decision-3 direction. Commit e813a84c “refactor: restore default error stack rendering” on feature/BE-3657-phase3 (the MR !585 head at the time, since superseded by 4df2562a), pushed + ls-remote verified.

• error.stack is now the crate’s default indented tree via the ErrorCode debug hook: ASCII charset, no color, full-path codes per frame. (The debug-hook plumbing 1f9cd611 had deleted as dead - install_debug_hook, Charset/ColorMode, HOOK_INIT - is restored.)
• The Datadog list-view newline-collapse quirk that MOTIVATED the custom renderer is now ACCEPTED as a display quirk, not worth a bespoke renderer.
• Momentary asymmetry at this commit (ELIMINATED the same day by 4df2562a, below): error.details[].code was still SHORT here (commit 4e519269), while the stack frames carried FULL-path codes. The short-code reversal below removed the split - every code-bearing field is full path now.
• The revert had conflicts against the three later branch commits, hand-resolved by a subagent.
• Gates green: clippy clean, 578 lib tests passed. Known-flaky scheduler timing test should_fire_inner_job_through_run double-fired once, then passed 3/3 in isolation.

Short codes reverted to full path (commit 4df2562a)

RE-REVERSES both short-code decisions from earlier in the program: the 2026-07-23 short error.code (Short error.code decision (2026-07-23), commit a58df341) AND the 2026-07-24 short error.details[].code (Short error.details codes (commit 4e519269), commit 4e519269). Andras reversed his own 2026-07-23/24 short-code direction on 2026-07-27: error.code and error.details[].code are FULL module path again, uniform with error.kind / error.type / error.errors / error.fingerprint. The short/full split is gone entirely - every code-bearing field now carries the same full-path form, which realigns with the ORIGINAL field-contract decision 2.

Commit 4df2562a “refactor: restore full path error codes” on feature/BE-3657-phase3, the new MR !585 head (pushed, ls-remote verified).

• Clean revert of a58df341 + 4e519269, plus ONE follow-up import fix.
• The short_error_code resolver (the PascalCase-segment heuristic) is DELETED along with its tests - zero references remain workspace-wide.
• Gates green: clippy clean, 575 lib tests passed / 0 filtered (down from the 577/579/580 counts because the short/full split-pinning tests went with the helper).

Consequences (these RE-REVISE the Pending DD cutover at BE-3657 merge checklist and reverse the 2026-07-23/24 shrink notes):

1. The @error.details.code facet is now FULL path - no longer a readable short form; the ::-quoting rule still applies to every code facet.
2. The legacy-monitor sweep is back to FULL impact: nothing in the new emissions resembles the legacy short codes anymore, so EVERY @error.code-keyed monitor/facet breaks format at cutover (the 2026-07-23 “shrinks the impact” note is reversed).
3. The “full-path-kills-mobile-readability” team-spec sentence is UN-WITHDRAWN: the 2026-07-23 decision had WITHDRAWN it (short codes had restored line-100 conformance); with full paths back, logs-and-apm.md line 100 (mobile readability) is violated again and the sentence needs RE-RAISING with the team. Back on the pending-team-sentences list.

MandoReport name FINAL (newtype + derive)

MandoReport is the FINAL name for BOTH the mando-core newtype (already shipped in !585’s seam-collapse signatures, commit f771805e) AND the derive #[derive(MandoReport)]. Chosen by Andras 2026-07-27 from the shortlist MandoReport / Trace / ErrChain. Zero rename churn since the newtype already carries the name.

derive(MandoReport) MR started, stacked off phase3

The #[derive(MandoReport)] MR (spec’d 2026-07-24, see Next MR designed: derive(MandoReport) (spec’d 2026-07-24)) is now IN PROGRESS: branch feature/BE-3657-derive cut off the phase-3 tip (worktree .worktrees/BE-3657-derive), NOT waiting for !585 to merge - the derive needs the phase-3 report API that only exists on that branch. The implementation plan is at docs/superpowers/plans/2026-07-27-be3657-report-derive.md (repo, untracked), updated this session for the new base and full-path code assertions.

• Rebased with the phase-3 tip: the branch had ZERO own commits, so it fast-forwarded cleanly, twice in one day - first e813a84c → 4df2562a (the short-code revert), then 4df2562a → ee84ea50 (the ebs upload_file restore from review round 1, see Review round 1 (2026-07-27, krisztian.fekete1)).
• Transplant plan at !585 squash-merge switched to the BRANCH-NAME form: git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-3657-derive (previously pinned to the phase-3 tip sha 4df2562a). Naming the branch instead of a sha means future phase-3 tip moves no longer invalidate the recipe - exactly what the ebs revert did when it bumped the tip to ee84ea50. This lineage already survived rebase --onto twice with patch-id verification, so the transplant is low-risk.
• MR !585 through 2026-07-27: OPEN; review round 1 landed 3 comments from krisztian.fekete1 (see Review round 1 (2026-07-27, krisztian.fekete1)); not_approved is the only unmet merge gate.

Spec grounding audit (2026-07-27)

Before any derive code was written, the derive spec (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, untracked) was audited claim-by-claim against the branch at ee84ea50 and corrected. The pass caught several stale or wrong claims:

• Alias is MandoResult, not a Result shadow: the spec wrongly described a std-Result shadow named Result; the real alias shipped in the seam collapse (Native report signatures (commit f771805e)) is MandoResult<T, E>.
• The newtype is already SHIPPED (report.rs:15-59): MandoReport<C> landed with commit f771805e, so the mando-core work for the derive MR SHRINKS to the marker trait + a wrap_report helper + the boundary bounds - it is NOT a from-scratch newtype.
• Emitted paths are ::mando_core::report:: module paths: the crate root has ZERO re-exports and stays that way, so the derive expands to fully-qualified ::mando_core::report:: paths, not crate-root ones.
• “enum stays Clone / flatten keeps working” corrected: only 6 of the ~20 participating enums are Clone; the derive changes NO enum’s Clone status. The earlier framing implied a universal Clone that does not exist.
• North-star example dropped its wrong Clone derive: the real VolueEmsError is NOT Clone, so the spec’s exemplar was corrected to match.
• Rulings added: multiple #[report] attrs per variant are allowed; the from-scope ruling forbids from(reqwest::Error) (the only sanctioned sources are HttpRetryError + InvalidHeaderValue); derive-time error strings are FROZEN; clippy disallowed-methods is a NEW clippy.toml key with exactly 2 sanctioned #[allow] sites; the flip commit requires a test-enum marker; the thiserror 2.0.17-vs-2.0.19 spike gap is closed by a golden test; trybuild enters as a new =-pinned dev-dep with a Nexus-availability caveat.

A/B equivalence gate (new requirement, 2026-07-27)

New requirement from Andras (2026-07-27): the derive MUST produce BYTE-IDENTICAL emission to the verbose forms it replaces - it is a refactor, not a behavior change. The spec gained an “A/B equivalence gate” section and plan Task 5 grew a concrete suite:

• Per grammar arm (from-display, from-source-stored, wraps-message, wraps-unit, attr-chain): build the SAME failure through the verbose form AND the derived ? path.
• Assert equal: report_codes equal; report_details deep-equal after stripping per-level file/line; render_stack_tree equal after dropping location lines. Locations are asserted SEPARATELY per form.
• By-construction backing: the derive from() calls the same into_report the closures do, and the derive wraps() calls the same wrap_report that WrapExt::wrap delegates to - equivalence holds by shared code path, and the tests pin it.
• Failures are derive bugs, never assertion loosenings: if an A/B pair diverges, the derive is wrong; the test does not get relaxed.
• Per-adapter equivalence is carried by these pairs + the golden e2e wire gate; NO verbose twins are kept in adapters (the twins were already deleted by the seam collapse).

Status: spec and plan were aligned and grounded, then execution BEGAN - Tasks 1-3 landed review-approved before a design flaw forced the bridge-design pivot; see Derive execution and the bridge-design pivot (2026-07-27).

Derive execution and the bridge-design pivot (2026-07-27)

The most consequential move of the day. Subagent-driven execution of the derive plan STARTED on feature/BE-3657-derive; Tasks 1-3 landed review-approved, then Task 3’s implementer surfaced a design flaw the 2026-07-24 spike had masked, and Andras resolved it with the bridge design - a grammar + trait-layer change that makes cross-crate ? actually work.

The 2026-07-24 orphan-rule spike was wrong (masked by single-crate topology)

The original derive design assumed ? everywhere through From impls on the MandoReport newtype. The 2026-07-24 spike (scratchpad/spike-trackcaller) ran the newtype AND the enums in ONE crate, which MASKED Rust’s orphan rule. A 2026-07-27 controller spike (scratchpad/orphan-spike, two-crate topology) proved that in a downstream crate (mando-lib):

• impl From<ForeignSrc> for MandoReport<LocalEnum> is E0117 (orphan violation).
• impl From<MandoReport<A>> for MandoReport<B> is E0117 too.

Consequence: bare ? via From was IMPOSSIBLE for ALL wraps mappings (the auth chains) and for foreign sources like reqwest InvalidHeaderValue. Only crate-LOCAL sources (HttpRetryError) can ride bare ? via From.

Tasks 1-3 landed (commits 0a083ecf, c8c0370c, edf0dd32)

• Task 1 (0a083ecf): the MandoError marker trait + a wrap_report helper in mando-core; WrapExt delegates to it. Review approved.
• Task 2 (c8c0370c): bare #[derive(MandoReport)] marker emission + a trybuild compile-fail harness. trybuild enters as a NEW =1.0.118-pinned dev-dep. Review approved. Sanctioned collateral: a uuid serde-feature one-liner in mando-core/Cargo.toml fixing the known mando-core standalone-build bug (the pre-existing uuid/serde issue tracked since Build state (2026-07-22)).
• Task 3 (edf0dd32): #[report(from(SrcType))] From-impl generation with #[track_caller], 6 trybuild cases. Its implementer surfaced the orphan flaw above.

The bridge design (Andras, option A over local-only)

Andras chose the bridge (option A) over restricting the derive to crate-local sources only. mando-core gains two LOCAL traits the derive implements on the local enum (always orphan-legal, since the enum is the local type):

• MandoFrom<S> - maps a source S into this enum’s report.
• MandoWraps<I> - wraps an inner report I as a new chain level.

Plus two ext traits carrying the call-site methods: MappedExt / WrappedExt. Call-site DX:

• wraps sites collapse to .wrapped()?.
• foreign-source sites to .mapped()?.
• crate-local from() sites stay bare ?.

Grammar split (one route per source type per enum across from + mapped - a given source X cannot be on both):

• #[report(from(Src))] = LOCAL sources only (generates From, rides bare ?).
• NEW #[report(mapped(Src))] = ANY source (generates MandoFrom, rides .mapped()?).
• #[report(wraps(Inner, message))] = MandoWraps (rides .wrapped()?).

Verified compiling with correct ?-residual inference in the same spike.

Update 2026-07-28 (DX v2): this three-arm grammar is SUPERSEDED - from and mapped collapse into a single #[cause(Src1, Src2)] bare source list (bare ? for ALL sources via the now-coherent blanket From), so MappedExt / .mapped() are DELETED; MandoFrom survives as the blanket’s routing trait and wraps / .wrapped()? are unchanged. The whole #[derive(MandoReport)] is replaced by the #[mando_error] attribute macro. See DX overhaul v2: native MandoReport birth (2026-07-28).

For Agents: ext methods must not use map_err(fn-pointer)

The MappedExt / WrappedExt methods MUST use match + a DIRECT trait-fn call, never map_err(fn-pointer): fn-pointer coercion ERASES #[track_caller], degrading locations to the mapping body. This is the exact degradation the phase-3 reported() migration fixed (the 20 map_err(new_coded) sites, see Report vocabulary (commit f5beabd4)).

Spec and plan updated for the bridge

• Spec (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, untracked): spike finding 4 added, the bridge layer added to the architecture, grammar bullets updated, the north-star example now shows .wrapped()? / .mapped()?, the A/B pairs adjusted.
• Plan (docs/superpowers/plans/2026-07-27-be3657-report-derive.md, untracked): ruling 4b added; Task 4 rewritten to the bridge traits + wraps / mapped emission across TWO commits; the Task 5 A/B suite adjusted; the sweep tables now use mapped(InvalidHeaderValue) (the foreign source moved off from()).
• SDD ledger at .worktrees/BE-3657-derive/.superpowers/sdd/progress.md tracks all of it.

The branch now carries its first own commits (Tasks 1-3), so it is no longer the zero-commit fast-forward of the phase-3 tip described above; the branch-name transplant recipe still holds - git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-3657-derive replays the derive-only commits onto develop.

Derive MR complete and pushed (commit 574d757a, 2026-07-27)

The derive MR is IMPLEMENTATION-COMPLETE and pushed: feature/BE-3657-derive @ 574d757a, 16 commits on base ee84ea50 (ls-remote verified), still stacked on !585’s branch - it is NOT yet an MR (it waits for !585 to merge). Built subagent-driven: the full 14-task plan plus a 6-item fix wave, each task under an adversarial spec + quality review, then a whole-branch final review on Fable that returned “with fixes”; the 6-item fix wave addressed them and re-review approved with zero new issues. Tasks 1-3 (the marker + wrap_report + from-impl gen, documented above) are the first 3 of the 16.

What shipped (the bridge layer from the bridge design shipped as designed):

• mando-core: the MandoError marker + wrap_report helper + the bridge traits - MandoFrom / MandoWraps implemented on the local enum (always orphan-legal) plus the MappedExt / WrappedExt ext traits, all match-form preserving #[track_caller] (never map_err(fn-pointer)).
• The #[derive(MandoReport)] proc macro: the four grammars (bare marker / from / mapped / wraps), FROZEN derive-time error strings, and a 12-case trybuild UI suite (on the =1.0.118 dev-dep added in Task 2).
• Enforcement: MandoError trait bounds on the 4 boundary sites + a clippy Report::new ban with exactly 2 sanctioned #[allow]s.
• Six adapter sweep waves: ~50 call-site collapses (to bare ? / .mapped()? / .wrapped()?) plus a shared Utf8BodyError alias consolidated in util/http_client.rs.

Final gates at 574d757a: clippy clean; 586 passed / 0 failed / 32 ignored lib tests; all commits title-only conventional.

Tower-freeze law (the second execution discovery)

Sites inside raw-Report-typed tower fns cannot take the bridge collapses

The second big execution discovery, after the orphan-rule flaw that forced the bridge design. Where an enclosing tower fn is typed Error = Report<...> (raw, not MandoResult), a call site inside it cannot collapse to .wrapped()? / .mapped()? without an into_inner bridge, so its verbose auth ceremony stays. This is the derive-branch face of the walled-towers finding.

Auth-wrap score: 2 live .wrapped()? sites that could collapse now (metis_graphql query_events, microsoft send_chat_message) vs 5 frozen until the unwalling phase (ems, atp, mdr, opl, position_manager). The 5 frozen groups keep their verbose auth ceremony through this MR BY DESIGN.

Update (fix wave 86f3439f): the unwrap-after-bridge form (bridge-collapse then map_err(MandoReport::into_inner)), proven on metis get_token, could unfreeze all 5 in the unwalling phase - recorded as an option, not applied here - see Five-lens assurance pass + fix wave (2026-07-27, tip 86f3439f).

Verification stack (equivalence + locations + wire)

The derive is a refactor, so it was gated to BYTE-IDENTICAL emission (the A/B equivalence gate):

• A/B equivalence suite: 5 pairs (verbose form vs derived ? path), byte-identical emission per pair.
• Golden #[track_caller] pins for bare ? AND .mapped()? AND .wrapped()? - all three capture the exact call-site line on thiserror 2.0.17 (closing the 2.0.17-vs-2.0.19 spike gap flagged in the grounding audit).
• trybuild frozen strings: the 12 UI cases pin the derive-time error messages.
• e2e wire byte-compare on the local rig: PASS, byte-identical including locations. One apparent diff was root-caused to per-process HashMap iteration ordering in Volue data-group selection - environmental, not a build/derive difference.

Derive follow-up bundle and next steps

Recorded in the SDD ledger (.worktrees/BE-3657-derive/.superpowers/sdd/progress.md) for the ticket bundle, distinct from the phase-3 bundle:

1. The trybuild suite is NOT CI-gated - test.sh runs --lib only, so the proc-macro crate’s UI tests never compile in the pipeline (the same --lib blind spot as mando-ci-lib-only-test-gate-2026-07-22); ticket to add cargo test -p mando_flow_step_derive.
2. The clippy disallowed-macros / disallowed-methods bans are INERT under the workspace lint config - both the old tracing::error ban AND the new Report::new ban. clippy::all is allow and neither lint is cherry-pick-denied, so the bans never fire. The team should know the Report::new ban is documentation, not enforcement, until the lint is denied.
3. Derive UX minors: wraps message-on-unit ignored; wraps shape classification; enum-level attr ignored.
4. Supertrait-bound polish deferred to the unwalling phase.

Next: wait for !585 to merge, then transplant with the branch-name rebase (git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-3657-derive, the recipe), re-run the gates, and open the MR ONLY on Andras’s explicit yes.

Five-lens assurance pass + fix wave (2026-07-27, tip 86f3439f)

Andras-requested FULL review of the whole derive branch (ee84ea50..574d757a), run as FIVE parallel Opus review lenses, then a 3-commit fix wave for the actionable findings. Branch tip moved 574d757a → 86f3439f (19 commits total, pushed); still stacked on !585, still NOT yet an MR.

The five lenses (whole branch, both directions where a spec applies):

1. Repo-spec conformance (vs the derive spec + the parent 2026-07-15 design): CONFORMANT both directions, including a verified NO-reqwest-route (nothing rides from(reqwest::Error)) and the metis stored-source mapped() being spec-sanctioned.
2. Team-spec conformance (vs logs-and-apm.md): CONFORMANT on all 8 applicable construction rules - coded construction, no secret material, chain depth, #[track_caller], sensitive tokens retained, zero logging in construction.
3. Coding standards (vs AGENTS.md): CLEAN line-by-line, zero Important-or-above findings.
4. Adversarial bug hunt: NO CONFIRMED BUGS - all runtime behavior verified; only FIVE compile-error-only robustness gaps in derive attribute tolerance.
5. DX + SOLID/DRY: DX HOLDS - every happy path strictly better, escape hatches unchanged, frozen sites byte-identical; found two discoverability cliffs + one DRY defect.

The fix wave (commits b5ec99f7 / 81fade1a / 86f3439f, pushed) addressed lenses 4 and 5 only:

• (a) Derive parser hardening (the five compile-error-only gaps from lens 4): empty #[report()] / from() / mapped() now raise DERIVE-TIME errors, and wraps shape validation was added. 3 new frozen error strings + 3 new trybuild cases (UI suite 12 → 15).
• (b) Discoverability cliffs fixed (lens 5): #[diagnostic::on_unimplemented] on MandoFrom / MandoWraps, so a missing-route .mapped() / .wrapped() now tells the dev to declare the #[report] route (rendering scratch-proven on rustc 1.89); the derive doc one-liner now names the three route attributes (from / mapped / wraps).
• (c) Metis DRY defect fixed (lens 5): the FROZEN get_token helper now reuses the enum’s declared bridge routes via the unwrap-after-bridge form (.wrapped() / .mapped() then map_err(MandoReport::into_inner)), removing the twice-written auth message WITHOUT unwalling the tower fn. Byte-equivalent; the auth-level test stays green.

For Agents: unwrap-after-bridge could unfreeze ALL five walled auth towers

The fix-wave (c) form - call .wrapped() / .mapped() inside a raw-Report-typed tower fn, then map_err(MandoReport::into_inner) to flatten back to the tower’s fixed error type - is PROVEN viable on metis get_token. It could unfreeze ALL FIVE frozen tower auth sites (ems / atp / mdr / opl / position_manager) that carry declared routes (see Tower-freeze law (the second execution discovery)), collapsing their verbose auth ceremony onto the enum’s declared routes. Recorded as an OPTION for the tower-unwalling phase, deliberately NOT applied beyond metis in this MR.

Accepted without action (recorded so they do not resurface as findings): token-string type matching in the parser; a From-qualification hygiene nit; the shared from/mapped route rule (BY DESIGN, trybuild-pinned); the derive-vs-type name clash (MandoReport is BOTH the derive and the newtype - Andras’s named decision, see MandoReport name FINAL (newtype + derive)); the io::Error::new idiom in two tests; the A/B verbose-side location pin; A/B module placement in lib.rs; two team-spec latent notes (the derive PERMITS tuple shapes; markers on enums with pre-existing tuple/transparent variants); the opl/pm two-dialect note; and the py-wall marker-bound hole (TODO’d for the unwalling phase).

Spec sync: two sentences fixed in the repo derive spec (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, untracked) - the stale from(InvalidHeaderValue) wording (that foreign source rides mapped(), not from(), since the bridge pivot) and a stored-source exemption reconciliation.

Gates at 86f3439f: derive UI 15 cases, full lib 586 passed / 0 failed / 32 ignored, clippy clean.

Error postfix question resolved (*_reported gone, *_error_from_report by design)

Recorded so it does not recur: Andras asked whether the “_report function-name postfixes” had been removed. Verified on feature/BE-3657-phase3:

• All *_reported op-twin fns are GONE - the ~35 flat twins were deleted by the seam collapse (commit f771805e, see Native report signatures (commit f771805e)); adapters now expose ONE fn per operation.
• 13 *_error_from_report tower-flattening converters REMAIN by design - these are the walls (Key architectural finding: walled service towers (formal spec exception)): flat-typed consumers flatten reports at the fixed-error tower edges. Removal is deferred to the tower-unwalling / services phase, NOT this MR.
• The 5 dead conversion fns were already deleted on the branch (theme-2 precedent).

So the postfixes that survive (*_error_from_report) are intentional seam-flatteners, not leftover scaffolding.

Review round 1 (2026-07-27, krisztian.fekete1)

First review pass on MR !585, breaking the run of zero comments: 3 comments from krisztian.fekete1, all questions rather than change-requests. Two Andras answered in-thread; the third he agreed with and fixed on the branch.

1. mando-bess/src/lib.rs mock gate - the MANDO_DEBUG_MOCK_ERROR debug_error emitter in init_router: “do we want this in the production code?” Andras is pushing back in-thread: it is env-gated, default off, and exists as a Datadog-pipeline verification tool (the report-backed startup mock ported during phase 3). This was always a known veto path - it was flagged-default 2 in the phase-3 plan (see Flagged defaults (Andras can veto before execution)). Unresolved pending the reviewer’s reply.
2. opl.rs create_orders deletion - “why was this deleted?” Answer (Andras in-thread): dead code, ZERO callers anywhere on develop (git grep verified; only self-references), removed in the opl sweep commit 64d04625 together with the OplOrder struct. A straight “delete unused code” precedent (review theme 2), not a behavior change. Update 2026-07-28: Andras REVERSED this stance and RESTORED the code rather than argue the deletion - commit 0427c68d re-adds OplOrder + create_orders (adapted to the current error-stack idiom, still dead code); phase-3 tip ee84ea50 → 0427c68d - see opl create_orders restored on !585 (commit 0427c68d).
3. ebs.rs upload_file restructure - “is this change really needed?” The reviewer was right: the SMB share / UNC path / user hoist was an OUT-OF-SCOPE mix-in (Andras in-thread: “its not in scope for this you are right, it got mixed in from the SMB fix branch”). Fixed on the branch - see below.

EBS upload_file restored (commit ee84ea50)

Commit ee84ea50 “refactor: restore ebs upload block layout” on feature/BE-3657-phase3 reverts the out-of-scope mix-in. upload_file’s diff vs develop is now ONLY the BE-3657 error-shape hunks (tuple to named variants + into_report) plus the MandoResult signature - the error-handling change and nothing else. Removed as parts of the same mixed-in SMB feature:

• the SMB share / UNC path / user hoist,
• the share / upload_path .attr() report enrichment, and
• the branch-added test upload_file_carries_samba_code_and_share_context (the ebs test module now matches develop exactly).

Gates green: clippy clean, 574 lib tests passed. Pushed, ls-remote verified; MR !585 head is now ee84ea50 (was 4df2562a).

For Agents

EBS errors no longer carry the share / upload_path ErrorAttr attributes after this revert. Re-adding them WITHOUT the block hoist is possible later if wanted: they are purely additive .attr() calls on the error path, independent of the SMB layout change.

The derive branch tracked this move: feature/BE-3657-derive (still zero own commits) fast-forwarded 4df2562a → ee84ea50, and its transplant recipe switched to the branch-name form git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-3657-derive so future tip moves no longer invalidate a pinned sha - see derive(MandoReport) MR started, stacked off phase3.

DX overhaul v2: native MandoReport birth (2026-07-28)

Second error-authoring DX overhaul, driven by Andras (continuation of the 2026-07-27 “Error hdl” session), parallel to the first DX overhaul. North star pushed all the way: bare ? for EVERY source regardless of source crate, and ONE annotation per error enum. Delivered as 4 subagent-implemented + reviewed commits on feature/BE-3657-derive, each wire-e2e-verified at the end, moving the tip [[#five-lens-assurance-pass—fix-wave-2026-07-27-tip-86f3439f|86f3439f]] → b7b04b71 (21 commits total on base ee84ea50, pushed + ls-remote verified). Still stacked on !585, still NOT yet an MR. The full design journey (3 decisions + 2 amendments) is recorded in the repo derive spec’s DX v2 section (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, untracked, never commit).

The design journey (3 decisions, 2 amendments)

1. Blanket From via MandoFrom (Andras’s ask: bare ? for ALL sources + single-annotation declaration). APPROVED, then hit E0119 (conflicting impls) against the silent Report-to-MandoReport lifting From (the DX-v1 ? auto-lift): the two blanket impls overlap in the open world.

   SECOND spike-masking incident: scratch crates lie about coherence scratchpad/orphan-spike) used a LOCAL FakeReport, which MASKED open-world coherence - the same class of false-green as the 2026-07-24 single-crate orphan spike. This is the SECOND spike-masking incident in two days (orphan rule discovered on 07-27, coherence on 07-28). Lesson (now twice-proven): verify orphan/coherence behavior in the REAL crate topology, never a scratch crate - scratch crates collapse the open world and hide E0117/E0119.

   The 2026-07-27 controller spike (

2. Amendment: explicit .lift() (approved as the E0119 escape - call .lift() at each seam instead of a blanket). Measurement then found 227 broken seams, not the estimated ~11.

   Lesson: implicit ? seams vastly outnumber visible .into() sites The ~11 estimate counted VISIBLE conversion points; the real count of places a source flows into a report through ? was 227. Implicit ? seams dwarf the explicit .into() / .map_err() sites you can eyeball, so an "add an explicit call everywhere" plan is far larger than it looks.

3. FINAL amendment: native MandoReport birth. into_report() / reported() now return MandoReport / MandoResult natively - the MandoError wall moves to CONSTRUCTION itself (the report is born mando-typed, not lifted from a raw Report at each seam). MandoReport’s std Error impl is REMOVED (the “anyhow trick”: like anyhow::Error and error-stack’s own Report, it no longer impls std::error::Error), which resolves the E0119 overlap and makes the blanket From<S: MandoFrom> COHERENT. Result: ~160 seams vanish (bare ? just works because the report is already mando-typed at construction), towers pay .into_inner() at ~40-50 sites, and .lift() / LiftExt survive only at tower-crossing edges.

What shipped (4 commits)

• 5acd801b - native birth: into_report() / reported() return MandoReport / MandoResult; the blanket From via MandoFrom; .mapped() / MappedExt DELETED (subsumed by the blanket + bare ?).
• 8ebd6ad9 - #[cause(Src1, Src2)] bare source lists REPLACE the #[report(from(...))] / #[report(mapped(...))] arms; the wraps keyword is kept; derive-time strings re-frozen.
• 4221f00b - #[mando_error] attribute macro: ONE annotation on the enum = thiserror::Error + Debug + the MandoError marker + cause routes, all at once. Shares a generate_routes engine with the derive path (one code path for both).
• b7b04b71 - 28-enum workspace sweep onto #[mando_error]; the #[derive(MandoReport)] export is DELETED; thiserror imports dropped in 20 files.

Attribute-ordering hazard empirically DISPROVED

The reviewer worried #[mando_error] (an attribute macro) plus the derives it generates might be order-sensitive. Proven false: attribute macros expand BEFORE derives regardless of source order, so #[mando_error] always sees the raw enum and the derives always see its output. No ordering constraint ships.

Authoring surface now

The full error-authoring API after DX v2

• Declare: #[mando_error] on the enum (that one line = Error + Debug + marker + routes).
• Route sources: #[cause(HttpRetryError, serde_json::Error)] per variant - a bare source list, any crate.
• Call sites: bare ? at EVERY call site regardless of source crate.
• Chains: .wrapped()? (the wraps route via WrappedExt).
• Escape hatches UNCHANGED: .wrap / .attr / into_report / .wrapped.
• Towers: still verbose until the tower-unwalling phase; they bridge with .into_inner() / .lift() at their fixed-error edges.

This SUPERSEDES the 2026-07-27 bridge grammar: from / mapped collapse into the single #[cause(...)] list, MappedExt / .mapped() are gone, and #[derive(MandoReport)] is replaced by #[mando_error]. The tower-freeze law still holds (walled towers stay verbose); DX v2 only changes their edge-bridging to .into_inner() / .lift().

Verification (byte-identical, wire-confirmed)

• A/B equivalence suite + 3 location goldens: byte-identical assertions throughout.
• Wire e2e vs [[#five-lens-assurance-pass—fix-wave-2026-07-27-tip-86f3439f|86f3439f]]: PASS - 4 event pairs, ALL contract fields equal including error.details file+line. The ONLY diff: 4 column-only location shifts at ts_data_retrieve.rs:102, mapping EXACTLY to the lift-form change (expected, not a regression). The known Volue data-group HashMap iteration nondeterminism was normalized again (environmental, as in the 07-27 compare).

KNOWN INTENDED DELTA (not drift)

mando-simulator’s init error! was switched to the report arm: it now GAINS errors[] / details[] and DROPS fingerprint on that ONE non-flow path. It was NOT byte-compared (it is a separate binary from the wire-compared mando-bess). Intended, recorded so it is not mistaken for a regression.

Status

DX v2 is DONE. The branch still waits on !585 merging before the transplant (git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-3657-derive, the branch-name recipe), a re-gate, and Andras’s explicit MR yes.

Update 2026-07-28 (later): the branch was RENAMED to feature/BE-4000 (own Jira key), the DX v2 delta passed a SECOND assurance pass (verdict CLEAN, tip now cd5d53a7 after a 2-commit standards fix-wave), and the transplant recipe is now git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000 - see BE-4000 rename, DX v2 assurance, opl restore (2026-07-28).

BE-4000 rename, DX v2 assurance, opl restore (2026-07-28)

Post-DX v2 events on 2026-07-28 (continuation of the “Error hdl” session): the derive branch got its own Jira key and was renamed, a SECOND assurance pass covered the DX v2 delta that the 07-27 five-lens pass predated, a 2-commit standards fix-wave landed the two standards findings (derive tip now cd5d53a7), and on the phase-3 side Andras RESTORED the opl create_orders dead code that review round 1 had questioned (phase-3 tip now 0427c68d, all 3 MR !585 comments resolved). The repo handover/spec/plan were updated in-repo this session (all untracked, never commit).

Derive branch renamed to BE-4000

The derive MR got its OWN Jira key, so the branch was renamed: feature/BE-3657-derive → feature/BE-4000 (old remote branch deleted, new branch pushed). BE-4000 is the FINAL derive-branch name.

Worktree DIRECTORY still says BE-3657-derive

The worktree directory stays .worktrees/BE-3657-derive (historical name kept); only the BRANCH was renamed, and the branch checked out there is now feature/BE-4000. Do not expect a directory rename.

• The transplant recipe is now git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000 (the branch-name form from derive(MandoReport) MR started, stacked off phase3, with the derive branch renamed). Every earlier feature/BE-3657-derive reference in this note is HISTORICAL; the live branch is feature/BE-4000.

Assurance pass 2 over the DX v2 delta (verdict CLEAN)

Andras-requested SECOND assurance pass, run because the 07-27 five-lens pass predated the DX v2 commits. THREE fresh Opus lenses over the v2 delta (86f3439f..cd5d53a7). Verdict: CLEAN on all six dimensions.

• Compiles.
• NO CONFIRMED BUGS - and the two riskiest claims were proven by a SCRATCH COMPILE, not just reasoning: (a) the reflexive-From disjointness (the blanket From<S: MandoFrom> does not overlap the reflexive/identity From) and (b) the attribute-macro expand-before-derives ordering (re-confirming the DX v2 finding). The native-birth tower into_inner/lift sites were verified LEVEL-PRESERVING, and #[track_caller] is intact.
• Spec-conformant.
• Testable: 585 lib + 15 trybuild + A/B 5 pairs + 3 location goldens.
• Lintable: clippy 0.
• SOLID/DRY holds: a single shared route-gen engine (the two generators collapsed into one, the DX v2 generate_routes engine).

Three latent no-instance edges documented (recorded so they do not resurface; none currently reachable):

1. last-path-segment Error/Debug dedup could drop a custom derive_more::Debug and silently change codes - but NO swept enum has one.
2. generic error enums are unsupported by route-gen - pre-existing, not introduced by DX v2.
3. hardcoded ::thiserror path in the generated code.

OPEN JUDGMENT CALL for Andras

An into_bare_report() helper would collapse ~52 into_report().into_inner() tower sites. Worth it ONLY if the unwalling phase is NOT the next MR - if unwalling IS next, those tower sites vanish anyway (the towers stop paying .into_inner()), so the helper would be churn. Deferred to Andras pending the phase ordering.

Standards fix-wave (commits 479cc312, cd5d53a7)

The two standards-lens findings from assurance pass 2, fixed in two commits:

• 479cc312 - scoped rustfmt collapse of the MandoFrom #[diagnostic::on_unimplemented] note that a DX v2 commit had left unformatted (scoped to the touched file, never bare cargo fmt).
• cd5d53a7 - rename the two “duplicate report mapping” derive-error strings to “duplicate cause mapping”, for consistency with the rest of the #[cause(...)] grammar vocabulary; 3 trybuild .stderr goldens regenerated to match.

Gates green: clippy 0, report_derive 8, lib 585/0. Pushed; BE-4000 tip is now cd5d53a7.

opl create_orders restored on !585 (commit 0427c68d)

On the PHASE-3 side (MR !585), reviewer krisztian.fekete1’s comment 2 (opl.rs create_orders, “why was this deleted?”) was resolved by Andras choosing to RESTORE the code rather than argue the dead-code deletion. Commit 0427c68d on feature/BE-3657-phase3 re-adds the OplOrder struct + create_orders fn.

• ADAPTED, not a raw revert: a straight revert would NOT compile - the original code referenced removed imports (VolueAtpOrderType, TimeDelta) and old tuple error variants. The restored form mirrors create_strategies exactly against the current error-stack idiom.
• Still dead code (no caller), per the reviewer’s request - a “restore what the reviewer asked about” move, not a re-wiring. This REVERSES the phase-3 opl sweep’s earlier deletion (commit 64d04625).
• Gates green. Phase-3 tip ee84ea50 → 0427c68d.

For Agents: the BE-4000 derive base is unaffected

The BE-4000 derive branch base STAYS ee84ea50 - the opl restore commit 0427c68d is NOT in the derive lineage. The transplant recipe (git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000) excludes the opl commit, so no re-sync of the derive branch is needed.

Status after 2026-07-28

• feature/BE-4000 is the FINAL derive branch; DX v2 is done and DOUBLE-ASSURED (the 07-27 five-lens + this 07-28 pass 2). Tip cd5d53a7.
• MR !585 (phase 3) is at 0427c68d with ALL 3 review-round-1 comments resolved: the mock gate answered in-thread, opl create_orders restored, the ebs upload_file mix-in reverted (see Review round 1 (2026-07-27, krisztian.fekete1)).
• Both await: !585 reviewer approval → merge → transplant (git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000) → Andras’s explicit MR yes for BE-4000. Update (rebase, same day): !585’s phase-3 tip moved 0427c68d → c3519e73 and the approval was reset - see MR !585 rebased onto develop, pipeline green (2026-07-28).

MR !585 rebased onto develop, pipeline green (2026-07-28)

The last phase-3 event of 2026-07-28 (continuation of the “Error hdl” session): feature/BE-3657-phase3 was REBASED onto develop 29ed8e34 to clear a merge conflict that opened when develop advanced 22 commits ahead of the branch’s base. Phase-3 tip 0427c68d → c3519e73, force-with-lease pushed. MR !585’s conflict is gone and the pipeline is SUCCESS on the rebased head, but the rebase RESET the approval (below), so the MR is not_approved and needs re-approval to merge.

The conflict and its semantic resolution

The rebase hit a 5-file textual collision: mando-bess/src/lib.rs plus 4 simulator files. Each was resolved SEMANTICALLY - keep develop’s functional changes, then re-apply phase-3’s error-stack shape on top - not a mechanical take-theirs / take-ours.

Landmine: a big-branch rebase can break the build with no textual conflict

NEW develop consumers of a migrated API break the build without conflicting

The consequential catch. While the branch sat in review, develop merged 2 NEW consumers of the very simulator-client API that phase-3 had migrated to the error-stack idiom: mando-lib/src/service/mando/asset_simulator.rs and mando-bess/src/live_simulator_init.rs. They call the pre-migration API signature, so after the rebase they failed to compile - yet neither was a TEXTUAL conflict (they are brand-new files develop added, no overlapping hunk), so conflict detection flagged nothing. Only a full workspace BUILD surfaced them.

Both were migrated to the phase-3 idiom simulator_client_error_from_report(e.as_report()) and the fix was folded into the commit that OWNS the signature change, so every commit still builds (the per-commit-buildable invariant, AGENTS.md §2.8) - the branch footprint grew 81 -> 83 files (the 2 new consumers).

General lesson: textual-conflict-free is NOT build-clean. A large-branch rebase can break the build through new develop code that consumes an API your branch changed; only building each commit catches it. Sibling of the cross-crate-fallout lesson (per-crate -p gates miss consumer crates; here it is per-rebase conflict detection missing new consumers).

Verification

• No conflict markers left anywhere.
• git range-diff vs the pre-rebase branch: 27=/3! - 27 commits identical, 3 changed (the commits carrying the semantic conflict resolutions plus the signature-owning commit that absorbed the 2 new consumers); all 30 commits preserved.
• Gates: 605 lib tests passed / 0 failed (higher than the pre-rebase 574/575 counts, since develop’s 22 commits added tests), clippy clean.

Consequence: the rebase reset the approval

GitLab drops approvals on ANY new commit

krisztian.fekete1 had APPROVED !585 after the round-1 comments were resolved. The rebase created new commit SHAs, and GitLab drops all approvals when new commits land - so !585 is back to not_approved. Net state: conflict GONE, pipeline SUCCESS, but the single remaining merge gate (not_approved) reopened; the MR needs RE-APPROVAL to merge. Record both as standing facts: (1) a big-branch rebase can break the build via new consumers with no textual conflict, and (2) any new commit (a rebase included) resets a GitLab approval.

The repo-side detail lives in a fresh handover docs/superpowers/HANDOVER-error-handling-2026-07-29.md (untracked, never commit), which SUPERSEDES the earlier HANDOVER-error-handling-2026-07-24.md; this note is the vault decision trail.

Local end-to-end verification (2026-07-23)

Full LOCAL end-to-end test of the phase 3 error_stack telemetry. Every emission below came from the real runtime, not tests.

Setup: phase-3 branch binary (feature/BE-3657-phase3 @ b394e0d4, rebase-descendant of the reviewed e0184bc3, built in .worktrees/BE-3657-phase3) run as a HOST process against the mando-cli local stack (mando up --datadog -p infra: postgres + wiremock + local-dd-agent). Env assembled from .cargo/config.toml.example with overrides; env file kept at the session scratchpad (mando-p3.env). Full run recipe + friction log: mando-local-host-run-recipe-2026-07-23.

Verified emissions

• Boot mock (MANDO_DEBUG_MOCK_ERROR=true): 4-level report; error.errors array of 4 full-path codes; error.details entries with real file:line AND attributes (data_group on the top level); error.stack box tree including the ErrorAttr sub-line and “1 additional opaque attachment” (the HttpContext); code == kind == type full path; root-cause error.message; trace correlation. (Both the box-tree stack shape and the opaque-attachment mention here are superseded 2026-07-24 - see error.stack Caused-by renderer decision (2026-07-24).)
• Volue EMS spot step (404 from unstubbed wiremock): single-level coded emission with fingerprint code|step_path (data_update.load_battery_momentary_data), file:line spot_data.rs:79, static-first boundary message.
• Volue EMS retrieve step: TWO-LEVEL chain in production: VolueEmsError::AuthenticationError + VolueEmsAuthenticationProviderError::FailedRequest; details entries at ems.rs:217 and ems_auth_provider.rs:155; chain-carrying boundary message Finished running step with errors: authentication error: '...': failed request (status: 404). This is the auth level the final-review fix wave added to the pilot retrieve path, confirmed live.
• Metis merit order step: full-path error.code (MetisError::AuthenticationError) with NO error.errors array - the documented walled-tower flattening (Key architectural finding: walled service towers (formal spec exception)), the mixed world working as designed.
• dd_formatter: renders error.errors/error.details as REAL nested JSON in the log lines (the shared error_json_field_value parse hook).

Datadog forwarding

12 error lines forwarded to Datadog EU via DIRECT INTAKE under env:dev-local-levander service:bess-os-service-mando. Host processes are not tailed by the local dd agent (its include filter scopes to mando-* containers, see mando-cli-datadog-local-logs-2026-07-22), so direct intake is the host-process path.

Session end state

Server left RUNNING at session end

localhost:8081, health 200, execution history queryable. Stack stops with mando down; the host process with pkill mando_bess.

Update 2026-07-24: this survived instance later held port 8081 and made a fresh launch die with AddrInUse while its old-format emissions kept flowing - see Datadog operational gotchas (2026-07-24).

DD_SERVICE env fix (commit ab8cfd11, 2026-07-22)

Rider commit on this branch replacing the hardcoded Datadog service name "bess-os-service-mando" in mando-lib with a DD_SERVICE env read, keeping the hardcoded value as fallback (also on empty string). It closes the mando-side root cause of the FR unsuffixed-service-name issue (fr-region-missing-datadog-logs-2026-07-21).

• Implementation: resolve_dd_service() + a OnceLock-cached dd_service() accessor in mando-lib/src/app/mod.rs (app feature gated).
• All 6 hardcode sites routed through it: the OTel Resource service name (get_tagging), the ddtags in dd_formatter.rs, and 4 raw global::meter("bess-os-service-mando") literals in service_base.rs (3x) and util/http_client_trace.rs that had bypassed the old const entirely.
• Inline test covers default / empty-string / override behavior.
• Deployment-neutral on the main stack: optimization-universe-iac terraform/bess_os_ecs.tf line 218 ALREADY sets DD_SERVICE = "bess-os-service-mando" on the mando container (algo containers and lambdas set their own names), so the code now honors what IaC says instead of ignoring it. The FR stack’s -fr-suffixed value starts winning once a build carrying this fix deploys there.

Dev prototype verification (build b7d15e2c)

Dev runs the throwaway build 1.16.1-feat.2682437891.b7d15e2c (branch feature/BE-3482-dev-deploy head b7d15e2c, static-first template merge; IaC pin applied 2026-07-16, Apply complete; NEVER merge, delete after the train lands). Stack = BE-3541 + BE-3656 + BE-3657 phases 1+2 + full-path codes + startup mock deep error + nested http.request/http.response + static-first boundary message + flow.step.system.

Verified in Datadog:

• Nested error.details: 4-level chain with real file:line values.
• error.errors array queries: element match, wildcard, and negation all work.
• Nested http shape: http.{request{method,url,version,body,content_length}, response{status_code,body,content_length}}.
• Trace correlation confirmed (closes the “verify dd.trace_id correlation” deferral); sample trace 0dc8d42739a08ac0805808b2d14b6644.

Mock still on the typed arm

The startup mock deep error fires once per task boot but still uses the typed error! arm with hardcoded JSON, NOT the report arm. Offer to upgrade it stands. Update 2026-07-23: superseded on the phase 3 branch - the mock is ported to mainline mando-bess as report-backed behind MANDO_DEBUG_MOCK_ERROR (default off). Effective once phase 3 merges; the throwaway dev-deploy branch keeps the old hardcoded-JSON form until then.

Pending DD cutover at BE-3657 merge

Datadog assets must be cut over when this deploys

All error.code / fingerprint values change format (short codes become full module paths) at the BE-3657 deploy. At merge time:

1. Create facets: @error.fingerprint, @error.errors, @error.details.code.
2. Inventory and update monitors keyed on SHORT codes, on flat http.* fields, and on the exact message "Failed running step".

Status 2026-07-23: !580 (phases 1+2) is MERGED, so this checklist becomes actionable with the first develop-based dev deploy carrying it.

Update 2026-07-23 evening: the short error.code decision (commit a58df341, in review with MR !585) shrinks the @error.code part of item 2: error.code stays Enum::Variant and closely resembles the legacy codes, so the format break narrows to error.kind/error.type, error.fingerprint, and the new array fields. NEW item 3: @error.source is RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message / @error.errors; one empty error.source consumer was found and fixed by Andras 2026-07-23.

Update 2026-07-24: error.details[].code also goes SHORT (commit 4e519269), so the @error.details.code facet keeps a readable short form; the full-path break remains on error.kind/error.type/error.fingerprint/error.errors. QUERY RULE for the new codes: facet values containing :: MUST be quoted (@error.code:"VolueEmsError::ApiError") - unquoted queries silently match nothing.

Update 2026-07-27 (short codes REVERSED): both short-code decisions are undone (commit 4df2562a) - error.code AND error.details.code are FULL module path again. This REVERSES the two shrink notes above: the format break is back to FULL impact across @error.code/@error.kind/@error.type/@error.fingerprint/@error.errors/@error.details.code (nothing resembles the legacy short codes now), and @error.details.code carries full paths. The ::-quoting rule still applies to every code facet. See Short codes reverted to full path (commit 4df2562a).

Relationship to prior direction

Supersedes the ErrorCode-derive / mando-lib-macro direction that only ever lived in worktree experiments (see Mando AGENTS.md Master Guide - there is NO ErrorCode derive on develop; error.kind/error.code are extracted at runtime by mando_core::error! from the Debug repr today). BE-3657 replaces that runtime Debug-repr extraction with static full-path codes captured at the error-construction site - originally via CodedExt, since 2026-07-24 via the ToReport report vocabulary (CodedExt is deleted, see Error-authoring DX overhaul (2026-07-24)).

Related

• BE-3656 APM Span Enrichment - immediate predecessor, merged as !578 (merge 4a0d297f); BE-3657 now sits directly on that merge, and the shared dev-deploy prototype validates the BE-3657 field shapes
• fr-region-missing-datadog-logs-2026-07-21 - the FR unsuffixed-service-name issue closed by the ab8cfd11 DD_SERVICE env fix on this branch
• mando-ci-lib-only-test-gate-2026-07-22 - CI gate parity discovery made while running this branch’s gates
• BE-3541 Single Error Emission - single boundary emission, merged as !571; precondition for the clean stacking
• BE-3482 Datadog Logs and APM Conformance - the conformance program this feeds
• BE-3613 Algo Services py-mando Conformance - sibling workstream moving the algo services onto the conformant wheel
• BE-3117 Per-Flow Error Context Store - the error-handling redesign lineage
• BE-4000 derive transplant - the #[mando_error] / #[cause(...)] / bare-? derive that replaced the seam-replication mechanics
• BE-4047 - the unwalling phase that began dismantling the frozen service towers
• BE-4067 whole-mando error unwalling - unwalling FINISHED (2026-08-04): zero production raw-Report left in mando-lib, the phase-3 “walled towers” finding fully dismantled
• mando-lib - crate hosting the migrated adapter/service layers; the report vocabulary (ToReport, MandoReport newtype) + extraction fns + macro arm live in mando-core
• Agent Context