Levandor

Flight execution_id Parity (BE-1595)

3 min read

Brings the Arrow Flight streaming path to parity with REST for execution-bound manual overrides, fixing a dev regression where enabling MANDO_FETCH_STRATEGY=flight hard-broke the optimization/forecast algo runners. Implemented 2026-06-17 on branch feature/BE-1595.

The regression

The algo runners (optimization, forecast) always fetch with an execution_id. When MANDO_FETCH_STRATEGY=flight was enabled in dev, every fetch hit a hard error in py-mando:

Flight strategy does not support access_token/headers/execution_id; use REST

The Flight client guard refused any request carrying an execution_id (or headers), so the entire runner workload was unusable on the Flight path.

Root cause

REST applies execution-bound manual overrides server-side: it looks up the flow execution by execution_id, then injects each manual_overrides entry into the matching DataPointFilters where manual_override.is_none(). This logic was inline in mando-lib/src/app/route/query_data_route.rs and existed only on the REST path. The Flight streaming path had no equivalent — so the client guard simply refused to send any execution_id rather than silently dropping overrides.

The fix (key design)

For Agents

Overrides are applied at the handler layer by mutating the DataPointFilters BEFORE streaming — NOT inside the streaming trait. retrieve_stream already honors DataPointFilter.manual_override identically to REST, so injecting into the filters is sufficient. This mirrors REST exactly and required no change to retrieve_stream.

1. Shared override helper (DRY)

Extracted the REST override block into a shared helper — single source of truth:

mando_lib::app::execution_override::apply_execution_overrides(
    &FlowRepository, Uuid, &mut [DataPointFilter]
)

plus a pure inject_overrides. Both REST get_data and the Flight server now call it.

2. execution_id on both tickets

Added execution_id: Option<Uuid> to both ticket types, each with #[serde(default)] for mixed-deploy safety:

  • client QueryTicket — mandarrow-client/src/ticket.rs (required adding the uuid dep to mandarrow-client)
  • server FlightTicket — mando-bess/src/flight.rs

3. Flight server applies overrides at the handler

do_get resolves ticket.execution_id and mutates ticket.data_points before retrieve_stream — exactly mirroring REST.

  • MandoFlightService gained an Arc<FlowRepository>, threaded through serve → spawn_flight_server.
  • lib.rs was restructured so each DB-mode branch builds flow_repository once and shares the Arc with both the flight server and get_app.

4. Relaxed client guard

py-mando/src/polars.rs fetch_with_strategy now errors only on a non-empty access_token. execution_id + headers no longer block the flight path.

  • headers is dropped on the flight path — cross-service trace stitching stays REST-only (deferred follow-up).
  • The access_token REST-fallback “safety net” is also deferred (still errors for now).

Correctness fact (verified)

Why handler-layer injection is enough

retrieve_stream honors DataPointFilter.manual_override identically to REST — same shared get_retrieve_sql + :manual_override binding for streamable DPs (repo.rs:497, repo_duckdb.rs:911). The materializable path literally calls retrieve. So injecting overrides into the filters at the handler layer produces identical results to REST.

Known limitation

DuckDB cache window vs. permanent storage

If a manual-override generation_time predates the DuckDB cache window for Permanent/DataPlatform-backed DPs, the streamed path returns MethodNotSupported where REST would still serve from permanent storage (repo_passthrough.rs:335). Fine for typical recent overrides.

Resolved 2026-06-24 — graceful REST fallback

This limitation was turned into graceful degradation in commit 7926278c: MethodNotSupported now maps to gRPC Unimplemented (was FailedPrecondition) and the client falls back to REST on is_unimplemented(). Forecast no longer crashes — it logs falling back to REST for the historical range. See be-1595-flight-rest-graceful-degradation-2026-06-24.

Deploy note

Not a mando-only deploy

The client changes live in the py-mando wheel. Deploying this requires rebuilding the mando image AND the wheel AND the optimization/forecast consumer images — not mando alone. See be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 for the consumer-repo CI angle and be-1595-enabling-arrow-flight-consumers-2026-06-16 for the per-consumer enablement decision.

Plan

Full plan on the branch: docs/superpowers/plans/2026-06-17-flight-execution-id-parity.md.

Graph View

Backlinks