Levandor

POST /data/insert wire format — Arrow IPC as a JSON u8 array, and why the CSV seed endpoint stays

5 min read

Research into whether the E2E harness could drop mando’s dev-gated CSV seed endpoint and insert test data through mando’s normal POST /data/insert from the client side. Answer: it is technically feasible — and rejected. This note records the wire format (which is unobvious and undocumented anywhere else), the proof that client-side construction works, and the drift argument that decided it.

Decision — keep the dev-gated CSV seed endpoint, reject client-side Arrow

Building the Arrow IPC body in mando-cli is possible (~250 LOC, ~17-20 new crates) and would remove the need for a mando-side test endpoint. It was rejected on drift risk: the body’s encoding is an undocumented internal of polars’ Deserialize impl that has already changed between polars versions, and no test in either repo would guard the client-producing side. The CSV endpoint’s contract is stable, server-side, and covered by 14 tests that get fixed alongside any polars bump.

The wire format

POST /data/insert — handler mando-lib/src/app/route/save_data_route.rs.

{
  "data_points": { "<data point id>": [80, 65, 82, 49, ...] },
  "update_info": null
}

The value per data point is an Arrow IPC byte stream serialized as a JSON array of decimal u8 values. Not base64. Not a binary body. A JSON array of numbers.

AspectFact
Payload typeArrow IPC STREAM format (not the file format — no ARROW1 magic/footer)
JSON encodingarray of decimal u8, e.g. [80,65,82,49,...]
Server-side readerpolars-arrow (the arrow2 fork) StreamReader
update_infooptional
Success204 No Content
Content-Typenever enforced — the handler takes body: String

Why a JSON number array and nothing else

The shape is not a design choice, it is a consequence of polars 0.49.1’s Deserialize for DataFrame. Its deserialize_map_bytes visitor accepts bytes through visit_seq only when driven by serde_json — a base64 string fails. So the only JSON form the server can parse is the element-by-element number array.

Fixture proof

mando-lib/test/adapter/mando/rest/simple-send-test-data.json — 936 IPC bytes expand to ~9 KB of JSON (the file is 8941 bytes on disk). That expansion ratio is the fingerprint of this format; if you see it, you are looking at an IPC-in-JSON body.

Required DataFrame schema

Taken from the sibling CSV route mando-lib/src/app/route/save_data_csv_route.rs, which builds the same DataFrame the IPC path produces:

ColumnTypeNotes
value_timeTimestamp(Microsecond, None)naive, UTC-meaning
generation_timeTimestamp(Microsecond, None)naive, UTC-meaning
fetch_timeTimestamp(Microsecond, None)naive, UTC-meaning
value (or value_x / value_y)Float64
idUtf8matrix and trade data points only

Rejected outright: a flag column, and any null cell.

Downstream, get_column_date_times ignores the timezone and normalises ns/us/ms, so microsecond-naive is the safe target rather than a hard requirement of the consumer.

Client-side construction is feasible — proven, not assumed

mando-repository/src/arrow.rs:59-88 (arrow_record_batch_to_polars_record_batch) already does exactly the round-trip in question: it writes with arrow-rs arrow_ipc::writer::StreamWriter and reads the result back with polars_arrow::io::ipc::read::StreamReader. Both sides are pinned together in mando’s Cargo.lock (arrow 56.2.0, polars =0.49.1), so arrow-rs stream output is known-good input for the exact reader /data/insert uses.

Costed options

OptionCostVerdict
arrow-ipc + arrow-array + arrow-schema in mando-cli~250 LOC, ~17-20 crates added to the distributed CLIfeasible, rejected (see below)
Depend on polars directly100+ cratesrejected — absurd for a CLI that ships as a small musl binary
Ship pre-serialized IPC fixturesnonerejected — the CSVs are the reviewable spec; opaque binary fixtures are not reviewable and not authorable by the team
Hand-roll the flatbufferhighrejected — untestable, silently goes stale

The deciding factor: drift risk

This is the whole argument, and it is a repeat of a failure this project already suffered.

  • The insert body’s encoding is not a published contract. It is an emergent property of polars’ internal Deserialize impl for DataFrame.
  • It already changed between =0.45.1 and =0.49.1.
  • If mando-cli produced the bytes, no test anywhere would guard the producing side — mando’s tests exercise its own reader, mando-cli’s tests would exercise its own writer, and nothing would put the two in the same process.
  • That is the same failure class as the DataPointId serde drift (see mando-cli-e2e-live-green-2026-08-05): unit tests stayed green on both sides for weeks while the live wire 400’d.

Against that, the CSV endpoint is:

  • a server-side contract, in the same repo as the polars dependency;
  • covered by 14 tests that a polars bump breaks loudly and locally;
  • expressed in the format the suites are already authored in.

Generalizable rule

Do not re-implement a wire format whose only specification is another crate’s private serde impl. If you must, the format’s owner and its producer have to live in the same test process — otherwise a dependency bump silently splits them.

Open improvement (not done)

The test endpoint is currently gated at runtime by MANDO_TEST_ENDPOINTS, meaning the code ships inside production binaries. It could be gated at compile time instead:

  • add a cargo feature test-endpoints;
  • #[cfg(feature = "test-endpoints")] on the module declaration and on the route registration;
  • the CI e2e image build adds --features test-endpoints.

Roughly 3 lines. The payoff is that test-only routes are absent from production binaries entirely rather than merely unreachable.

If client-side Arrow insert is ever revisited

The non-negotiable precondition is a contract test living in mando’s repo that round-trips an arrow-rs StreamWriter stream through serde_json::from_value::<DataFrame>. Without that single test, a polars bump breaks the CLI’s seeding with no local signal anywhere.

Minor finding

mando-cli seeds with Content-Type: text/csv. The server never reads it (the handler signature is body: String). The header is decorative — harmless, but do not treat it as part of the contract.

Graph View

Backlinks