Design for gating flow (and step) execution on declared external-service outages, using Maintenance/{country}/{Service} interval-double datapoints plus a FlowGuard. The central result: the whole gate reduces to a single bitmask AND, because each service contributes exactly one bit, not a set of accepted values.
Model
Each external system gets an interval-double datapoint at Maintenance/{country}/{Service} (see the maintenance-datapoint column in External Integrations). Its value over an interval is a severity level:
| Value | Name | Meaning |
|---|---|---|
| 0 | Normal | System healthy, no special handling |
| 1 | Warning | Failures are expected but the system may work |
| 2 | Outage | Assume full unavailability |
Flow steps declare their dependencies in the flow YAML under external:, optionally with transient: true for a dependency whose failure the step tolerates.
- name: fetch_order_book
type: VolueAtpOrderBook
external:
- Maintenance/FI/VolueAtp
- service: Maintenance/FI/OPL
transient: trueCore insight: it is one bit, not a set
The per-service “accepted values” set is only ever {0,1} or {0,1,2}. A declaration either tolerates an outage or it does not. Severity 1 never blocks anything, so it drops out of the gate arithmetic entirely and exists purely for reporting and operator context.
So per service, per flow, there is exactly one boolean: does this flow require the service to be below outage level?
Requirement predicate
For service s in a flow:
R(s) = OR over all declarations of s of ( propagates_failure(step) AND NOT transient )
The min over per-declaration thresholds that a naive design would use collapses to a plain OR, because there are only two effective levels. If any non-transient declaration on a failure-propagating step exists, the service is required.
propagates_failuremust fold up the ancestor chainA hard step nested under a parent with
failure_status: Warningcontributes nothing: its failure is absorbed before it reaches the flow result.propagates_failure(step)therefore has to walk the ancestor chain, not just look at the step itself. Skipping this makes the guard stricter than the runtime, blocking flows that would actually have completed. This is the one case worth a dedicated test.
For Agents
transient: trueandfailure_status: Warningare mathematically identical at the gate: both make a declaration non-blocking. They differ only downstream (transient still runs the step and downgrades the error; Warning lets the step fail cheaply). Do not build two code paths in the guard for them.
Formula
Assign each service a fixed bit index i(s) at codegen time.
required = fold over s: acc | (R(s) << i(s)) // static per flow, codegen-emitted const
outages = fold over s: acc | (O(s) << i(s)) // dynamic
where O(s) = value(Maintenance/{country}/{s}, now) == 2
blockers = required & outages
can_run = blockers == 0
No branch is needed inside the fold: R and O are 0/1, so the shift places the bit and a false predicate contributes 0, which is the OR identity.
OR is idempotent and associative, so the “aggregate and dedup across the tree” step the design doc calls for is structural: fold the WorkflowStep tree with |. No HashSet, no dedup pass.
Cost at runtime: one AND and one compare, O(1). A u64 covers 64 services; 13 exist today.
The same mask can be evaluated per step, giving per-step skipping for free with identical code.
Reporting
blockers.count_ones()gives how many services are blocking.- Iterating the set bits of
blockersnames them (bit index maps back to theExternalServiceenum variant). - The resulting error variant maps to HTTP 423, alongside the existing
FlowDisabled/SystemDisabledarms.
Worked example
Verifying the formula against the design doc’s second matrix. Bit assignment: VolueAtp=1, PositionManager=2, Metis=4, OPL=8, MDR=16.
| Step | Declares | Contributes |
|---|---|---|
| A | VolueAtp, OPL (transient) | VolueAtp |
| B | all transient | nothing |
| C | under failure_status: Warning | nothing |
| D | Metis, PositionManager, MDR (transient) | Metis + PositionManager |
required = 0b00111
VolueAtp / PositionManager / Metis are capped at severity 1; OPL and MDR are unconstrained. This matches the design doc’s table exactly.
Placement
| Piece | Crate | Why |
|---|---|---|
ExternalService, ServiceMask, ExternalGate | mando-core | Pure primitives, zero I/O, and mando-core must stay lightweight for the Python extension |
FlowGuard (reads the interval doubles) | mando-lib/src/workflow/ | Needs repository access, lives with the rest of the workflow engine |
The ExternalService enum should be codegen output, generated from the same YAML that declares the maintenance datapoints, so that the datapoint path and the bit index stay in sync by construction rather than by discipline.
Runtime note
Batch the 13 interval-double reads into one repository call, not 13. The gate itself is free; the reads are the only cost.
Open items
- The YAML needs both the bare-string form (
external: [Maintenance/FI/VolueAtp]) and the struct form (- service: ..., transient: true) to deserialize, soExternalDependencyneeds an untagged two-variant enum. - System strings in
#[step(system = "...")]are not normalized today (see the warning in External Integrations); if the gate ever keys off them rather than off explicitexternal:declarations, they must be normalized first.