Levandor

py-mando Wheel ↔ Mando Server Version Skew (Flight wire types) — BE-1595

3 min read

The #1 deploy hazard for the Arrow Flight feature: the py-mando wheel and the mando server are deployed independently, but they share wire types over Flight + REST. A develop merge that bumped a shared struct broke old-wheel ↔ new-server. The durable fix is to make every boundary wire type forward/backward tolerant with serde.

This is the single most likely thing to break a deploy

The wheel (mandarrow-client Flight wire types + the bess client Python API, e.g. BessOptClient.send(update_id=...)) ships on its own cadence from the mando server. Any change to a struct that crosses the wire — Flight ticket/metadata OR REST request/response — can break a consumer running an older wheel against a newer server (or vice versa).

The incident

A develop merge bumped a shared DataPointUpdateInfo:

  • renamed fetch_time → update_time
  • added update_id

This broke old-wheel ↔ new-server: the old wheel could not deserialize the new shape, and the send side surfaced as:

send() got an unexpected keyword argument 'update_id'

Root cause

The wire types were not tolerant of additive/renamed fields. An older deserializer encountering a renamed field (or a stricter struct rejecting unknown fields) fails the whole payload.

Fix (commit 73cbdcf3) — wire-type tolerance rules

Make every Flight + REST boundary struct tolerant:

Ruleserde attributeWhy
Every optional / added field defaults#[serde(default)]A newer field absent in an old payload deserializes fine
Renames keep the old name readable#[serde(alias = "old_name")]Old payloads with the old field name still deserialize
Never reject unknown fields on boundary structsNO #[serde(deny_unknown_fields)]A newer field present in a new payload doesn’t blow up an old deserializer

Shipped with compat tests: old-shape JSON must still deserialize into the current struct. These tests are the regression guard — they pin the contract that an old payload remains readable.

Going forward

For Agents — wire-type change checklist

When touching ANY struct that crosses Flight or REST:

  1. New fields → #[serde(default)].
  2. Renames → keep #[serde(alias = "old")] on the new name.
  3. Never add #[serde(deny_unknown_fields)] to a boundary struct.
  4. Add/extend a compat test that deserializes the old JSON shape. With these, additive changes are non-breaking in both directions.

The two hard constraints that remain

  1. You cannot retroactively fix already-released versions. Tolerance only helps versions built after the fix. If an old wheel is already in the wild without the alias/default, it stays broken against the new shape. So: when a wire type changes, deploy mando + the consumer wheel from the same build.
  2. The send-side mismatch is not serde-fixable. send() got an unexpected keyword argument 'update_id' is a Python API signature mismatch in the bess client, not a deserialization issue. It only resolves by rebuilding the consumers on the matching wheel — there is no compatibility shim for a method signature.

Graph View

Backlinks