Arrow Flight Gap Analysis 2026-04-30

Arrow Flight Gap Analysis

Branch: feature/BE-1595 | Date: 2026-04-30

Status

Code-complete across 5 layers (DataPointStream, StreamingRepository, StreamingMandoService, MandoFlightService, mandarrow-client). Not production-ready — critical wiring and hardening gaps remain.

Critical Gaps

Gap	Location	Impact
Flight only starts in DuckDB mode	lib.rs:176-188	Not spawned in Hybrid (prod) or SQLite (staging)
No graceful shutdown	lib.rs:183	Fire-and-forget tokio::spawn, no JoinHandle
Port 50051 not in service manifests	bess-service.yaml, mando.service.yaml	Unreachable in containers, no health check
Infinite recursion on circular calculated DPs	service.rs:1051	No visited set, stack overflow risk
Orphaned spawn_blocking tasks	stream.rs:129-156	Timeout drops receiver but task holds connection

High Severity

• Schema timeout returns empty stream, not error (stream.rs:148-154)
• Topo sort silently drops cyclic DPs (service.rs:1118-1120)
• make_retrieve_stream materializes all DPs before streaming (service.rs:971-991)
• Client dp_id_array.value(0) panics on null (mandarrow-client/stream.rs:25)
• Silent channel failures via let _ = (repo_duckdb.rs:851-867)
• No timeout on DuckDB query/batch iteration (repo_duckdb.rs:857-868)

Medium Severity

• No FlightTicket input validation (flight.rs:66-68)
• dp_id column collision not checked (flight.rs:98-101)
• No retry/reconnection in client (client.rs:35-43)
• Error context lost — all tonic Status wrapped to generic ClientError (client.rs:40)
• Env var docs say MANDO_FLIGHT_PORT, code uses FLIGHT_PORT

Testing

• 0/124 test cases from FLIGHT_TEST_PLAN.md executed
• 1 integration test exists (ticket roundtrip only)
• No multi-DP demux tests, no REST parity validation

What’s Solid

• Feature flag wiring correct and consistent
• Dependency versions pinned and aligned
• CI builds with --features flight, tests with --all-features
• DuckDB pool shared via Arc between REST and Flight
• DataPointStream abstraction well-designed with unit tests
• mandarrow-client crate well-organized

Minimum to Test on Dev

1. Add Flight spawn to Hybrid mode in lib.rs
2. Add port 50051 to service manifests
3. Push and deploy
4. Use mandarrow-client or grpcurl to verify