Levandor

BE-3117 Per-Flow Error Context Store

5 min read

Per flow run, emit exactly one aggregate business ERROR log line that retains 100% of that run’s error data. Replaces the prior “OEF v1” approach (committed on the same branch) which demoted in-flow ERROR→WARN in the dd_formatter and emitted a thin repo-read aggregate — that lost fidelity and coupled on magic strings. All mechanism lives in mando-lib.

For Agents

Mechanism = a task-local Vec<ErrorRecord> store created inside the flow’s spawned task, written via the record_error! macro at error sites, drained and emitted as one tracing::error! after the flow scope returns. The invariant is record exactly once per flow, gated by ErrorWithStepStatus.logged_at_site. The dd_formatter ERROR→WARN demotion from OEF v1 was REMOVED.

Goal

One aggregate business ERROR per flow run, with full fidelity:

  • Carries every error’s kind/code, full anyhow cause chain, HTTP context, validation results, call site, timestamp — not a generic wrapper string.
  • No dd_formatter log-level rewriting and no magic-string coupling (the two failure modes of OEF v1).
  • This is “OEF v2” — it supersedes the OEF v1 that was committed earlier on feature/BE-3117.

Mechanism

ErrorRecord

mando-lib/src/tracing/tracing.rs. Clone + a hand-written serde::Serialize impl that redacts HTTP request/response bodies when http_context.sensitive (auth/credential payloads). Fields:

  • timestamp, step, error_kind, error_code
  • error_source — full anyhow {:#} cause chain
  • message, http_context, validation_results, event_type, call_site, extra

FLOW_ERROR_CONTEXT

A tokio::task_local! of Arc<Mutex<Vec<ErrorRecord>>>, sibling to the existing TASK_CONTEXT task-local.

record_error! macro

mando-lib/src/tracing/error.rs, #[macro_export]. Behaviour:

  • In a flow scope → builds an ErrorRecord (kind/code resolved via mando_core::error::kind_code), pushes it into the store, and emits an INFO breadcrumb that NEVER contains raw bodies.
  • Out of scope → falls back to mando_core::error! (an ERROR).

Macro gotcha: TT-muncher required

Implemented as a TT-muncher (__record_error_parse! / __record_error_emit!) because the naive $(...)? optional-named-field form hits a local ambiguity macro error. Don’t “simplify” it back to $(...)?.

Why the macro lives in mando-lib, not mando-core

record_error! cannot live in mando-core because mando-core must not depend on mando-lib, and the macro needs ErrorRecord / FLOW_ERROR_CONTEXT / HttpErrorContext — all mando-lib types. It calls mando_core::error::kind_code + mando_core::error!, both of which all caller crates already depend on.

Flow boundary

mando-lib/src/workflow/flow.rs. The store is created inside the flow’s tokio::spawned task — task-locals do not propagate into a detached spawn (the same span-inheritance class of bug documented in BE-1842 Datadog Observability for tokio::spawn). Scoped as:

FLOW_ERROR_CONTEXT.scope(store, TASK_CONTEXT.scope(ctx, service.oneshot(param)).instrument(span))

After the scope returns (outside the flow span), flush_flow_errors(trace_id, &store, completed) -> bool drains the store (mem::take) and emits one tracing::error! with flat facets plus a full JSON payload:

  • flow.summary=true
  • flow.exec_id
  • flow.error.count
  • flow.error.codes
  • flow.error.failed_steps
  • event_type=Integration
  • flow.errors — full JSON payload of all ErrorRecords

FlowCompletionGuard holds the same Arc and flushes on Drop (with completed=false) to cover panic/cancel. No double emission: guaranteed by mem::take + the completed flag + an empty-store early-return in flush_flow_errors.

Step level

StepResult::log’s Err arm now records (in-scope) via a local record_step_error! that uses ErrorRecord::from(&ErrorWithStepStatus) to reuse the already-computed kind/code rather than recompute them. Out of scope it falls back to the existing step_error! (ERROR). The record path is gated on !logged_at_site.

Why From<&ErrorWithStepStatus> lives in workflow/mod.rs

From<&ErrorWithStepStatus> for ErrorRecord is defined in mando-lib/src/workflow/mod.rs because that struct’s error_kind / error_code / http_context fields are module-private.

The “record exactly once” invariant

The core design rule: each error is recorded exactly once per flow, gated by ErrorWithStepStatus.logged_at_site.

  • A site that records itself (via record_error!) MUST return an error with logged_at_site = true — constructed via message_logged / mark_logged — so the step’s log() takes the WARN path and does not re-record.
  • Errors not logged at their site are recorded by the step-level path, which fires only when !logged_at_site.

For Agents

This logged_at_site flag is the evolution of the logged_at_failure_site flag described in flow-step-log-message-dropped-2026-05-26 and the c945514e/4c543cb1 BE-2272 follow-ups. Same semantics (“I already logged myself, parent dedupe to WARN”), now also driving whether the step-level path records into the store.

Gotcha found & fixed: pre-existing MarketNotFound double-log

The 4 MarketNotFound step sites had a pre-existing double-log (independent of this feature):

  • intraday/v2 — energy_bids_step, open_position_notification_step
  • manual_schedule/v3 — energy_bids_step, open_position_notification_step

Each site error!’d the error AND the step step_error!’d the same error — both at ERROR. Fixed by adding ErrorWithStepStatus::mark_logged() so the returned error sets logged_at_site = true, collapsing the two into one record.

Datadog-facing change

The aggregate’s failed-steps facet was renamed flow.failed_steps → flow.error.failed_steps, introducing a new flat flow.error.* hierarchy (flow.error.count, flow.error.codes, flow.error.failed_steps). event_type=Integration was preserved. Continues the root-flattening convention from BE-2272 — dashboards referencing @flow.failed_steps need a column rename.

Status

Implemented on branch feature/BE-3117 via subagent-driven TDD (two-stage review per task).

  • All green: mando_core + mando_lib + mando_bess tests pass, py_mando compiles, clippy clean on touched files.
  • NOT yet committed (awaiting user).
  • Replaces the committed OEF v1 on the same branch.

Known non-blocking follow-up

No end-to-end failing-flow integration test through the real spawned task. The flush / guard / record paths are covered by unit tests plus a flush-payload retention/redaction test, but not exercised through the actual tokio::spawned flow task end-to-end.

Scope of the "one ERROR per flow" guarantee

The guarantee holds for business / step errors. Three out-of-scope infra error! calls can co-emit by design: DelayCalculationFailed, cancel_queued_steps, PostFlowServiceFailed.

File:Symbol Index

ConcernLocation
ErrorRecord struct + hand-written redacting Serializemando-lib/src/tracing/tracing.rs
FLOW_ERROR_CONTEXT task-localmando-lib/src/tracing/tracing.rs
record_error! + __record_error_parse! / __record_error_emit!mando-lib/src/tracing/error.rs
Flow scope wiring + flush_flow_errors + FlowCompletionGuardmando-lib/src/workflow/flow.rs
From<&ErrorWithStepStatus> for ErrorRecordmando-lib/src/workflow/mod.rs
Step-level record_step_error! / StepResult::log Err armmando-lib/src/workflow/mod.rs
mark_logged / message_logged / logged_at_sitemando-lib/src/workflow/mod.rs
kind_code / error! fallbackmando-core::error
  • flow-step-log-message-dropped-2026-05-26 — the logged_at_failure_site/message-drop bug this feature’s invariant evolves from; logged_at_site is its successor flag.
  • BE-2272 — DD root-field flattening; this adds the flow.error.* flat hierarchy in that style.
  • BE-1842 Datadog Observability — documents the tokio::spawn context-inheritance class of bug; same reason the store is created inside the spawned task.
  • mando-lib — crate hosting ErrorRecord, FLOW_ERROR_CONTEXT, the macro, the flow boundary.
  • mando-bess — REST API where flows are triggered.
  • Agent Context — dense agent reference.
  • LOG — daily activity log.

Graph View

Backlinks