Levandor

Enabling the Arrow Flight Client Path in BESS Consumers (BE-1595)

3 min read

How to actually turn ON the Arrow Flight client path in the BESS Python consumers — the runtime switch, the silent-fallback gotcha, and the per-service ECS topology that decides whether enabling is a one-line env flip or real network plumbing.

The runtime switch

py-mando selects Flight vs REST at fetch time via an env var:

VarDefaultNotes
MANDO_FETCH_STRATEGYrestFlight used only when value == flight (case-insensitive); anything else → REST
MANDO_FLIGHT_HOSTlocalhost
MANDO_FLIGHT_PORT50051
MANDO_FLIGHT_PROTOCOLhttp
  • Read in py-mando/src/flight.rs:16 (struct MandoFlightConfig, envconfig).
  • Fetch path reads it per-fetch at py-mando/src/polars.rs:224-236.
  • Whole path is behind #[cfg(feature = "flight")]. The published dev wheel IS compiled with -F flight, so enablement is purely the runtime env var — no rebuild needed.

CRITICAL gotcha — silent REST fallback

Absence of errors does NOT prove Flight is being used

If a Flight fetch errors at runtime, py-mando silently falls back to REST (py-mando/src/polars.rs:234). After enabling, you must confirm via logs/metrics that the Flight path is actually exercised — a “clean” run may be silently on REST the whole time.

A corollary: setting MANDO_FETCH_STRATEGY=flight on a service that can’t actually reach the gRPC port (e.g. default localhost where mando isn’t local) is a noisy no-op — it fails every fetch and falls back to REST.

Per-service ECS topology (optimization-universe-iac, terraform)

The right action depends entirely on whether the consumer is co-located with mando in the same ECS task.

Co-located → just add 3 env vars (DONE)

bess-os-algo-optimization and bess-os-algo-forecast run in the same ECS task as bess-os-service-mando (bess_os_ecs.tf), reaching mando over localhost. Enabling Flight = add:

MANDO_FETCH_STRATEGY = flight
MANDO_FLIGHT_HOST     = localhost
MANDO_FLIGHT_PORT     = tostring(var.mando_flight_port)

Done on branch feature/mando-arrow (commit d3db63e).

Separate task → NOT an env flip (left on REST)

bess-os-dashboard-trader is a separate ECS task (trader_dashboard_ecs.tf) that reaches mando over the network domain (MANDO_REST_HOST = local.mando_domain, HTTPS).

No network path to the Flight port from the dashboard

The Flight gRPC port (50051) is only a host port inside the bess-os task — there is NO NLB / target-group / SG path for the dashboard to reach it. Enabling Flight here is not an env-var flip; it needs gRPC-over-network plumbing (NLB, or an ALB HTTP/2 gRPC target group + SG rule). Left on REST for now. Setting the var with the default localhost host would just fail-and-fallback to REST (noisy no-op).

Server side (already in place)

The mando container exposes the Flight gRPC port via var.mando_flight_port (default 50051):

  • Set as MANDO_FLIGHT_PORT on the mando container + gRPC port mapping (bess_os_ecs.tf ~211-213, ~307).
  • SG rule in security_group.tf.

For Agents

Decision rule for “enable Flight on consumer X”: (1) is X in the same ECS task as bess-os-service-mando (bess_os_ecs.tf)? → add the 3 env vars with host localhost. (2) Is X a separate task (its own *_ecs.tf, talks to mando over local.mando_domain)? → do NOT just set the env var; you need a gRPC NLB/ALB-HTTP2 target group + SG rule first, else it silently falls back to REST. Always verify Flight is actually exercised (logs/metrics) — silent REST fallback at polars.rs:234 hides misconfig.

Graph View

Backlinks