Levandor

mando-cli — two-repo data-level E2E harness (`mando e2e run` / `mando verify`)

5 min read

A data-level E2E harness spanning TWO repos on the same branch name poc/e2e-tests: /Volumes/bandi/coding/poc/mando-cli (the runner) and /Volumes/bandi/coding/poc/mando (a dev-gated CSV insert endpoint). Suites are expressed in the team’s existing test-data taxonomy, so a suite is authored as CSVs rather than as code.

Superseded 2026-08-05 evening — harness is GREEN 8/8 live

The harness has now been run against a live stack and passed 8/8, twice, from wiped volumes — see mando-cli-e2e-live-green-2026-08-05. The two warnings below are historical; both are resolved. Note that the “only a live run catches mando-side schema drift” prediction fired verbatim: upstream rewrote DataPointId serde to a bare string and only the live run saw the 400.

Historical status (2026-08-04): never executed against a live stack

Both branches are UNCOMMITTED and the harness has NEVER been run end-to-end. Gates below are compile/test/clippy only. Two live-only risks remain: the datapoints query body is now hand-mirrored in mando-cli’s tests (see Critical 1), so only a live run catches mando-side schema drift; and mando’s branch base is bugfix/BE-3333, 130 commits behind develop (rebase decision open).

Update 2026-08-05 — mando side rebased and pushed

The rebase decision is resolved: mando’s poc/e2e-tests was rebased onto origin/develop (~135 commits of drift), is now 3 ahead / 0 behind, and is committed and pushed — the CSV insert endpoint is no longer untracked-only. Full record, including the two rebase-forced fixes and the Apple Silicon PyO3 link gotcha: mando-e2e-rebase-2026-08-05. The mando-cli half is likewise fully committed and pushed (8 commits), pending merge via the approval-gated MR !5 (see mando-cli-gitlab-release-flow-2026-08-05), and the live harness re-verify against the rebased mando is in progress today — the “never run end-to-end” warning above is superseded.

Suite layout

test_set_N/
  input/     system/<sys>/*.csv     bess-os/<dp>.csv
  expected/  system/<sys>/*.csv     bess-os/<dp>.csv

mando e2e run <suite> phases, in order:

  1. wipe — down -v
  2. up — service_env + forced MANDO_TEST_ENDPOINTS
  3. ready — REST poll, 60s budget
  4. migrate
  5. seed
  6. per set — mock reset → stubs → seed → flow run --bundle → verify

mando verify <expect.yaml> runs the same assertions standalone. Sections: flow, mock, logs, spans, datapoints, outbound.

Server side (mando repo)

New dev-gated POST /data/insert/csv/{*datapoint_id}, mounted only when MANDO_TEST_ENDPOINTS=true.

Why a new endpoint was unavoidable

The normal insert body carries a polars DataFrame as Arrow IPC bytes, so a CSV→insert path is impossible to build client-side. The new endpoint round-trips the query CSV format, including colonless offsets.

e2e/test_set_1 uses data-update v2 — chosen because its per-step run toggles and reference_date-derived timestamps make expected values derivable rather than guessed.

mando query removed

Deleted entirely (nobody used it). With it went the mandarrow-client / mando-core / arrow-array worktree dependencies, so cargo build --all-features compiles clean for the first time in weeks.

Two criticals caught by review

Critical 1 — the self-confirming-test trap

The datapoints query body was wrong on three counts: id must be an object {id, id_fragments} (not a bare string), range is REQUIRED and internally tagged {"type":"Fix", from, to}, and reference_date must be RFC3339 (not the literal "now"). The reason review caught it and the tests did not: the unit tests asserted the invented shape against itself — they serialized the wrong struct and compared it to the same wrong expectation, so they passed while describing a body the server would reject. Fix: push the body through serde against a structural mirror of the real server types, with ground truth taken from an existing seed_data.py. Generalizable lesson: a test that round-trips your own model proves only self-consistency. Any wire-format test needs ground truth from outside the code under test.

Obsoleted 2026-08-05: the {id, id_fragments} object shape described above is no longer correct — upstream rewrote DataPointId’s Deserialize/Serialize to a bare JSON string, and the mirror tests stayed green while the live query 400’d. Current shape and fix: mando-cli-e2e-live-green-2026-08-05 (MR !6).

Critical 2 — the runner's own stack had zero WireMock mappings

The infra template mounts only __files/, never mappings/ — so no stub was ever loaded into the runner’s WireMock. The EMS adapter’s auth POST went unmatched and no suite could ever have passed. Fix: an input/system/_stubs/*.json verbatim-mapping hatch. Premise corrected during review: the EMS refresher does NOT fire at boot (run_periodically discards the first tick; 55-minute interval). The auth stub is needed for the lazy login inside the flow, not for a boot-time refresh — the original justification was wrong even though the fix was right.

Other notable fixes

  • service_env overlay could wedge every compose command. If the overlay named a service absent from the run, every compose invocation broke and only mando up could recover.
  • wipe destroyed volumes before validation. Suite validation that rejects a bad suite ran after down -v, so an invalid suite still cost you the stack.
  • Adapter endpoints take bare host:port + a separate _PROTOCOL var. Only VOLUE_EMS_PROTOCOL defaults to http; METIS, POSITION_MANAGER, and MDR default to https. The scheme-bearing form was wrong and had already propagated into SKILL.md and the guide prose — both corrected.

Gate

RepoResult
mando-cli682 lib + 8 integration; clippy --all-features zero; --all-features build clean
mando370 tests; clippy clean on touched files

Graph View

Backlinks