Levandor

"E2E Data Suite Linux Dev" GREEN in real GitLab CI — first ever

5 min read

The e2e harness now runs inside mando’s own GitLab pipeline and passed for the first time. This closes the last unproven seam of the harness design: mando e2e run --external-stack against CI-provided services: instead of a locally-orchestrated compose stack.

MILESTONE — pipeline 2735150017, job runtime 72s

E2E Data Suite Linux Dev GREEN in real CI. Suite output: “all assertions passed”, junit uploaded. --external-stack is no longer an unproven code path.

What the green job actually proves

The whole chain ran unassisted, end to end:

LinkProven
Cross-project artifact fetchthe released mando-cli binary is downloaded from the mando-cli project into the mando pipeline via CI_JOB_TOKEN + a job-token allowlist entry
Sidecar servicespostgres:17 and wiremock/wiremock:3x as GitLab services:
Shared network namespaceservices share the job’s netns, so WireMock is reachable on localhost:8081 (not a service hostname)
Host-process servicemando_bess started as a background host process from the pipeline’s own build artifact — not a container
Migrationsrefinery migrations applied on boot against the postgres sidecar
Suitetest_set_1 passed: all assertions passed
Reportingjunit uploaded to the job

For Agents

This is the CI counterpart of the local run in mando-cli-e2e-live-green-2026-08-05. Local = mando-cli orchestrates compose; CI = GitLab provides the stack and mando-cli runs with --external-stack. Both paths are now verified green on the same suite.

Boot-env whack-a-mole — and the rule that ends it

Getting mando_bess to boot in CI cost one 15-minute pipeline per missing env var, discovered one at a time.

Wart 1 — DataPlatformConfig reads its own disable flag too late

DataPlatformConfig::init_from_env().unwrap() runs before DATA_PLATFORM_DISABLED is read. So the “disabled” path still requires FINGRID_DATABASE and OUTPUT_LOCATION to be present, or the service panics at boot.

Design wart to raise with the mando team

A disable flag that is evaluated after the config it disables has already unwrap()ed is a boot-order bug, not a config requirement. Until it is fixed, “disabled” subsystems still need their vars set to dummy values.

Wart 2 — FINGRID_API_KEY panic

Next pipeline: panic at fingrid.rs:29 on a missing FINGRID_API_KEY. Same shape — a hard unwrap at construction rather than a lazy/optional read.

The lesson (do this instead)

Canonical known-good env set for mando_bess

/Volumes/bandi/coding/poc/compose.override.yml is the reference environment. Diff against it rather than deriving the required set from the code — code-derived guessing is exactly what turned this into N sequential 15-minute pipelines.

The final missing set turned out to be exactly:

  • FINGRID_API_KEY
  • AFRR_AUCTION_RESULT_DEADLINE
  • FCR_AUCTION_RESULT_DEADLINE
  • BATTERY_STATIC_DATA_MDR_PATH

Branch rules — poc/* now runs the dev pipeline

mando’s .branch_rules:dev gained poc/* (a one-line change), so poc/e2e-tests gets the full dev pipeline including the new e2e job. See Branch Strategy for the pre-existing pattern set (feature/*, bugfix/*, rc/*, develop).

A premature feature/e2e-tests mirror branch — created only to get a pipeline before the branch rule existed — was deleted.

Deleting a branch mid-pipeline kills its late jobs

The mirror branch’s half-finished pipeline failed on a missing ref: jobs that had not yet started could no longer git fetch the deleted branch. If you need the pipeline results, let it finish (or cancel it explicitly) before deleting the ref.

Dependency-cache fix — chef cooked the wrong feature set

Committed in container.linux.chef.build.Dockerfile.

The cargo-chef layer cooked dependencies with a bare --release, while the actual jobs build with different feature sets:

ConsumerFeatures
chef cook (before)(none) --release
.gitlab/scripts/build.sh--features flight
.gitlab/scripts/test.sh--all-features

Cargo keys its cache per feature set, so every job missed the cooked cache and recompiled arrow-flight / tonic from scratch. The Dockerfile now cooks both variants.

The benefit does not land until this reaches develop

The container rebake trigger is develop-only, so branch pipelines keep using the old image. Measure the win after merge, not before.

Escalation options if it is still slow

  • Scheduled weekly rebake — keeps the cooked layer fresh against Cargo.lock drift.
  • sccache + S3 — compilation cache independent of the image layer.
  • Not an option: GitLab cache: — it cannot hold /init/chef/cook/target, which lives outside the project directory (GitLab cache paths must be relative to CI_PROJECT_DIR).

The real pipeline whale is elsewhere

PyMando Win Dev at ~2656s dominates wall-clock regardless of the Rust dependency cache. Optimising the chef layer does not touch it.

Known improvement — junit granularity

SHIPPED 2026-08-06 — 9c39a88, pushed to main + release, released

--junit now emits one testcase per assertion, classed <test set>.<section> over flow / mock / logs / datapoints / spans / outbound, with human-readable stable names — step load_battery_timeseries is Success, no unmatched requests, POST /ExternalData/DataGroups at most 0x. mando verify gained --junit too. Verified in mando pipeline 2735668937: the Tests tab shows 8 named cases, 0 failed under suite E2E Data Suite Linux Dev. +20 unit tests (1632 total). Full reasoning, the dual-label pattern, and the “case names must come from the expectation, not the result” GitLab constraint: mando-cli-junit-per-assertion-2026-08-06.

Previously the whole test_set_1 reported as one junit case, so the MR widget could only say “the suite failed” — it now names the assertion that broke.

(The artifacts: when: always + reports: junit: convention itself is inherited from bess-os-ci-components’s python-test.)

Graph View

Backlinks