Levandor

Flight → REST Graceful Degradation (F4 capability-gap fallback) — BE-1595

3 min read

Forecast crashed on the Flight path because a historical fetch that predates the DuckDB cache window returns RepoError::MethodNotSupported, which originally mapped to gRPC FailedPrecondition and was treated as fatal by the py-mando client. The fix makes Flight degrade gracefully: stream what the cache window can serve, fall back to REST for ranges that predate it.

For Agents

This is the resolution of the “Known limit” flagged in be-1595-flight-execution-id-parity-2026-06-17 — manual-override / historical reads predating the DuckDB cache window returning MethodNotSupported from repo_passthrough.rs:335. That note diagnosed it; this note fixes it (commit 7926278c).

The capability gap

The Arrow Flight passthrough / DuckDB repo can only stream data within its cache window. A retrieve_stream over a range that predates the cache — i.e. data that lives only in Permanent storage — cannot be served by the streaming path and returns RepoError::MethodNotSupported (repo_passthrough.rs:335). REST does not have this limitation: it serves the same range from permanent storage.

So this is a genuine capability gap, not an error: Flight can’t do historical, REST can. The right behavior is to fall back, not to fail.

Symptoms

  • Optimization worked fully on Flight (it only fetches within the cache window).
  • Forecast crashed when MANDO_FETCH_STRATEGY=flight — it fetches historical data for its lookback, which predates the cache.

Root cause (the fatal classification)

Two pieces conspired to turn a recoverable capability gap into a fatal error:

  1. Server side — repo_error_to_status (mando-bess/src/flight.rs) mapped RepoError::MethodNotSupported → gRPC FailedPrecondition.
  2. Client side — the py-mando client only fell back to REST when the error was is_retryable() (transport-level errors). A FailedPrecondition is not retryable, so it propagated as Fatal and crashed forecast.

Fix (commit 7926278c)

Reclassify the capability gap so the client fallback recognizes it:

LayerFileChange
Servermando-bess/src/flight.rs (repo_error_to_status)Map RepoError::MethodNotSupported → gRPC Unimplemented (was FailedPrecondition)
Client errormandarrow-client/src/error.rsAdd ClientError::is_unimplemented()
Client fallbackpy-mando/src/polars.rsFall back to REST on `is_retryable()

Keep CyclicDependency as FailedPrecondition

Do NOT blanket-remap everything to Unimplemented. RepoError::CyclicDependency is a real error that REST cannot fix either — it must stay FailedPrecondition so it propagates as fatal. Only the capability gap (MethodNotSupported = “this transport can’t serve this range”) becomes Unimplemented. The semantic line: Unimplemented = “wrong transport, try the other one”; FailedPrecondition = “the request itself is bad, no transport will help”.

Net behavior after the fix

  • Flight streams everything it can (cache-window fetches).
  • For ranges that predate the cache, the server returns gRPC Unimplemented, the client recognizes it via is_unimplemented(), and falls back to REST for that fetch.
  • Optimization: works fully on Flight (cache-window fetches only — never hits the gap).
  • Forecast: logs falling back to REST (gRPC Unimplemented) then completes successfully.

How to verify in logs

A healthy forecast run on Flight will emit a falling back to REST log line (gRPC Unimplemented) for the historical portion, NOT a crash. Seeing that line is the expected signal that graceful degradation is working — distinct from the silent REST fallback on transport errors.

Graph View

Backlinks