Levandor

Fatal steps log the same error twice (leaf boundary + ancestor wrapper boundary)

3 min read

A Fatal flow step emits its ERROR line once at its own step boundary and again at every ancestor wrapper node, because status_or_error() returns Err for Fatal and the ? in the call macros re-surfaces the identical ErrorWithStepStatus at the parent. Verified 2026-09-02 against mando origin/develop tip b5770766 and prod tag v1.17.1+2757320894.7c1ef44f (7c1ef44f).

Still live on develop

git diff --name-only 92213459 85452c3e -- mando-lib/src/workflow/ is empty, so MR !601 (BE-4067) never touched the emission path. This is NOT the BE-3541 bug (detection site + boundary); this is boundary + ancestor boundary, a different axis, and it survived the single-emission migration.

Symptoms (prod Datadog)

One incident, exec_id 5937b3e7-d39f-44d0-bdd0-fd8dc721b217, produced two ERROR events from the same emission site mando-lib/src/workflow/mod.rs:361, differing only in step path:

Step pathflow.step.execution_timeFingerprint
data_update.load_battery_timeseries1.809s...InternalDuckDbError|data_update.load_battery_timeseries
data_update5.783s...InternalDuckDbError|data_update

Because error.fingerprint embeds the step path, the two copies do not aggregate: one root cause shows up as two distinct fingerprints in Datadog.

Timing confirmation that the second event is the wrapper waiting on its slowest child: tokio::join! makes the wrapper finish with the slowest sibling; log gap 3.969s vs exec_time gap 3.974s, a match within 5ms.

Root cause

Three pieces, all in mando-lib/src/workflow/:

  1. run_step (mod.rs:465-540) calls state.log() unconditionally at line 499 for every step it executes, wrapper nodes included.
  2. call_parallel! (macros.rs, @unwrap arm at line 120) runs $step?.status_or_error()?.
  3. status_or_error() (mod.rs:382-388) returns Ok(status) when is_error_or_better() (Skip / Success / Warning / Error), but Err(err) for Fatal, returning the identical ErrorWithStepStatus.
pub fn status_or_error(self) -> Result<StepStatus, ErrorWithStepStatus> {
    match self.result {
        Ok(result) => Ok(result.step_status.clone()),
        Err(err) if err.status.is_error_or_better() => Ok(err.status),
        Err(err) => Err(err),
    }
}

The ? propagates the Fatal error out of the generated flow’s call(), so the wrapper step (flow: Wrapper, the desc on WorkflowStep::SequentialSteps) finishes with the same error object and re-logs it through the same line 499.

For Agents

Blast radius facts:

  • Fires only for Fatal. Error status is swallowed into Ok by status_or_error() and never propagates, so it emits once.
  • Copy count = 1 leaf + one per ancestor node carrying a desc. Deeper flow trees emit more than 2 copies.
  • NOT parallel-group specific: the sequential call macros (macros.rs lines 11, 20, 34, 45) use the same status_or_error()?.

Fix constraints

The Fatal → Err propagation is intentional and pinned by two tests:

  • status_or_error_preserves_error_status_as_ok (mod.rs:998)
  • status_or_error_propagates_fatal_as_err (mod.rs:1005)

So the fix belongs at the emission decision, not at the propagation. It must also not reintroduce a logged_at_site-style flag, since BE-3541 deliberately removed exactly that mechanism.

Suggested shape: only WorkflowStep::Step emits the error payload; wrapper nodes (SequentialSteps / parallel group descriptors) emit status only.

Underlying trigger in that prod log: an already-fixed bug

The failure being double-logged was a DuckDB duplicate primary key on (data_point_id, value_time, generation_time).

  • That is precisely what MR !610 (BE-2132) fixes via dab6cba6: it replaces the hardcoded (data_point_id, value_time) dedup grouping/join with the full schema.pk_columns.
  • git merge-base --is-ancestor confirms dab6cba6 and f418e9b3 are NOT in prod 7c1ef44f, but ARE on develop and on the test image 336ce49f.

Prod is hitting an already-fixed bug

The duplicate-PK crash in prod is fixed on develop/test but not in the deployed prod build. The double-logging defect is separate and unfixed everywhere.

Graph View

Backlinks