Levandor

E2E harness first live GREEN (8/8) — and the three upstream drifts it caught

4 min read

The two-repo E2E harness was run against a live stack for the first time and came back GREEN 8/8, twice, from wiped volumes, against the rebased mando branch. Getting there required fixing three separate upstream drifts that no unit test in either repo could have caught, plus one upstream behavior change that needs to go back to the mando team.

Status

mando e2e run verified GREEN 8/8 on two consecutive runs from wiped volumes. mando side at 2af0c6c9 (branch head after the history rewrite). mando-cli fix is MR !6 on bugfix/e2e-datapoint-id-string.

Follow-up the same evening — GREEN in real CI too

The suite then passed inside mando’s GitLab pipeline (E2E Data Suite Linux Dev, pipeline 2735150017, 72s), running --external-stack against postgres:17 + wiremock/wiremock:3x sidecars with mando_bess as a background host process. Full record: mando-e2e-ci-green-2026-08-05. Both execution modes — locally-orchestrated compose and CI-provided stack — are now verified on the same suite.

Drift 1 — DataPointId serde became a bare string (the predicted trap, fired verbatim)

The 08-04 handover warned that mando-cli’s datapoints query body is hand-mirrored from mando’s types, so only a live run could catch schema drift. That is exactly what happened.

Was{"id": {"id": "...", "id_fragments": [...]}} — a struct
Now"bess/..." — a bare JSON string (upstream rewrote both Deserialize and Serialize)
Symptom400 from POST /data/query/v2
Unit testsstayed green throughout

Fix (mando-cli): src/e2e/datapoints.rs, data_point_id() now emits json!(dp_id) instead of the two-field object. Four tests updated, including the serde-mirror test query_body_deserializes_as_the_service_would_read_it — its Filter.id is now a String.

The self-confirming-test trap, second act

This is the same failure mode catalogued as Critical 1 in mando-cli-e2e-harness-2026-08-04: the mirror test round-trips mando-cli’s own model, so it proves self-consistency and nothing else. The 08-04 note predicted the trap in writing; it still fired, because a hand-mirrored wire type has no mechanism to notice upstream moving. A live run is the only gate on this seam. Any future change to the mirror must be validated against a running service, never against cargo test.

Drift 2 + 3 — two env vars in mando’s e2e/suite.yaml

Both fixed on the mando side (e2e/suite.yaml, branch head 2af0c6c9).

VarDriftConsequence if wrong
MANDO_SETUP_ACTIVE_VERSION=V1_4now REQUIREDservice crashloops at boot without it
MANDO_FLOW_SCHEDULER_DISABLEDrenamed to MANDO_SETUP_SCHEDULER_DISABLEDold name is silently dead — the 0 5 * * * * data-update cron fires mid-suite and breaks unmatched_max: 0

The rename is silent, and it breaks the suite non-deterministically

Nothing rejects the obsolete MANDO_FLOW_SCHEDULER_DISABLED; the scheduler simply stays enabled. The failure then shows up as an unrelated-looking mock assertion (unmatched_max: 0 violated by cron-triggered outbound calls) at whatever point in the run the cron happens to land. Same MANDO_FLOW_* → MANDO_SETUP_* prefix migration already recorded in mando-local-host-run-recipe-2026-07-23 — the .cargo/config.toml.example staleness and this suite staleness are the same upstream move.

New upstream behavior change — DataPointId::new rejects non-alphanumeric fragments

Raise with the mando team

DataPointId::new now rejects any path fragment that is not purely ASCII alphanumeric. Ids with an underscore inside a segment — e.g. bess/fi_north/soc_state, and the whole in_ / out_ corpus naming convention — now return 400 on query.

This is a behavior change, not a harness bug. Open questions for the team: is the restriction intended, and if so what is the migration path for existing ids that carry underscores?

Doc drift to fix on our side: mando-cli’s docs/e2e-guide.md documents underscore-bearing ids as valid examples. That guidance is now wrong and must be corrected once the upstream intent is confirmed.

Graph View

Backlinks