Levandor

E2E flow coverage — the suite reports which flows it exercises (and unit coverage was rejected)

4 min read

mando e2e run now ends by reporting how many of the service’s flows the suite actually exercised, and that number — not line coverage — is what mando’s GitLab pipeline parses as its coverage metric. mando-cli main 5a135b8; mando poc/e2e-tests 96b5b89c. Extends the CI work in mando-e2e-ci-green-2026-08-05 and mando-cli-junit-per-assertion-2026-08-06.

For Agents

Coverage here means flows exercised / flows the service declares, never lines or branches. If asked to “add coverage” to mando’s e2e pipeline, the answer is a new e2e test set covering an uncovered flow — not a coverage instrumentation tool.

The decision — flow coverage, not unit coverage

An interim cargo-llvm-cov unit-coverage job was built, ran green, and reported 40.36%. It was then fully reverted by user decision: the number people want from this pipeline is which product behaviours the e2e suite proves, not what fraction of Rust lines a unit suite touches.

Do not reintroduce llvm-cov here

The revert was deliberate and complete. A future “let’s also add line coverage” would put two competing numbers on the same pipeline and the coverage: key only takes one.

What the runner prints

At the end of a suite run:

flow not covered: as-auction-update
flow not covered: auction
flow not covered: intraday
flow not covered: manual-schedule
flow coverage: 1/5 flows (20.0%)
  • The denominator comes from the live service flow inventory — the same endpoint backing mando flow list — so it tracks whatever the service under test actually declares, with no list to maintain in the suite.
  • If the inventory fetch fails, the runner warns and skips the report. It never fails the suite: coverage reporting is observability, not an assertion. A missing inventory must not turn a green functional suite red.
  • 36 new unit tests came with it (1686 total in mando-cli).

The GitLab side

mando 96b5b89c adds to the E2E Data Suite Linux Dev job:

coverage: '/flow coverage: \d+\/\d+ flows \((\d+\.\d+)%\)/'

GitLab takes the first capture group of the last match on job stdout as the job’s coverage value; that is what feeds the MR widget and the badge.

The format and the regex are one contract

The regex requires a decimal digit (\d+\.\d+). Printing a bare 20% would match nothing and GitLab would silently show no coverage — an empty coverage value produces no error, exactly like the empty-dotenv trap in mando-cli-gitlab-release-flow-2026-08-05. Any change to the summary line has to be made in both repos together.

Badge URL (job name is URL-encoded, spaces as +):

/badges/poc%2Fe2e-tests/coverage.svg?job=E2E+Data+Suite+Linux+Dev

First live result — pipeline 2736559170

20.0% = 1/5 flows.

Flows
Covereddata-update
Uncoveredas-auction-update, auction, intraday, manual-schedule

That is the honest starting point and the backlog in one line: four flows have no e2e set at all. The 8/8 assertion green from mando-cli-e2e-live-green-2026-08-05 was always about one flow — this metric is what made that scope visible.

Same-day companions

  • Test set renamed test_set_1 → data-update-beskar-soc. Sets are now named after the flow plus the scenario, which is what makes the coverage report readable.

    The rename cost nothing in JUnit history — by design fabc589), classnames derive from the case's flow: field, not its directory name. Pipelines 2735716225 (before the rename) and 2736559170 (after) both emit data-update.flow / .mock / .logs / .datapoints. The directory name appears only as a collision-disambiguation fallback when two cases target the same flow. So no (classname, name) pair moved, nothing was orphaned, and there was no Tests-tab reset. This is the stable-label rule from mando-cli-junit-per-assertion-2026-08-06 holding under a rename — and it is why case directories can be renamed freely.

    Since the flow-scope change (mando-cli

    This supersedes that note’s classname table, which still documents the pre-fabc589 <test set>.<section> scheme.

  • mando-cli docs/ is now untracked and gitignored — those guides are AI-internal working material, not shipped documentation.

    Supersedes an earlier pointer mando-cli-junit-per-assertion-2026-08-06 deliberately deferred user-facing JUnit reference to docs/e2e-guide.md "in the repo". That file is no longer in git. Treat repo docs/ as local-only; anything that must survive belongs in this vault.

  • Agent skill gained an e2e case-anatomy section (.claude/skills/mando/SKILL.md) so an agent can author a case.yaml — the practical prerequisite for anyone raising that 20%.

Graph View

Backlinks