Levandor

mando-cli JUnit — one testcase per assertion (and why case labels must come from the expectation)

6 min read

mando e2e run --junit and the new mando verify --junit now emit one JUnit testcase per assertion instead of one per test set, so GitLab’s MR widget names the assertion that broke rather than the suite that contained it. Branch feature/junit-per-assertion, commit 9c39a88, repo /Volumes/bandi/coding/poc/mando-cli. This closes the open improvement flagged in Known improvement — junit granularity.

For Agents

User-facing reference (flags, classname scheme, sample XML) lives in the repo at docs/e2e-guide.md → “JUnit report — --junit <path>”. This note deliberately does not duplicate it; it records the reasoning, the invariants, and the traps.

Shipped and verified in CI — 2026-08-06

9c39a88 was pushed directly to main and release (per user instruction) and released through the release branch. Confirmed live in mando pipeline 2735668937: the GitLab Tests tab renders 8 named cases, 0 failed for suite E2E Data Suite Linux Dev — sample names step load_battery_timeseries is Success, no unmatched requests, POST /ExternalData/DataGroups at most 0x. The change added 20 unit tests (1632 in the suite).

The fix was a plumbing fix, not new evaluation logic

The most useful thing to know before touching this area: the verify engine was already per-assertion. src/e2e/verify.rs evaluates every expectation separately and builds a Vec<Assertion>. The report collapsed for two purely mechanical reasons:

WhereWhat it didWhat it does now
verify::run (src/e2e/verify.rs)discarded the Vec<Assertion> and returned a bare boolreturns the assertions it evaluated
junit_cases (src/commands/e2e.rs)mapped one SetResult → one TestCaseflat-maps each set’s assertions into cases

Generalizable

When a report is coarser than the engine behind it, check whether the granularity was thrown away at a boundary before assuming it was never computed. Here the whole feature was “stop returning bool”.

The dual-label pattern

Every Assertion now carries two names, constructed together by a Label type in src/e2e/verify.rs:

pub struct Label {
    pub section: &'static str,
    pub name: String,   // terminal
    pub case: String,   // JUnit
}
  • name — the machine label the terminal prints, e.g. mock.requests[GET /path].
  • case — the human sentence JUnit names the test with, e.g. GET /path at least 1x.

They are separate on purpose: the terminal label must stay byte-identical for humans reading a run and diffing output, while the JUnit label answers to GitLab’s history-tracking rules (below). Collapsing them into one string would force one consumer’s stability requirement onto the other.

Label::dotted(section, suffix, case) is the helper for the common section.suffix terminal form.

The non-obvious constraint — case names come from the EXPECTATION, never the result

The rule

A JUnit case name must be derivable before the assertion is evaluated. Never build it from the observed value.

GitLab attaches a test’s history to the pair (classname, name). If the label changed between a passing run and a failing run, GitLab does not see a pass → fail transition — it sees one test disappearing and a different test appearing. The MR widget’s “newly failed / previously failed” classification, and the test’s flakiness history, both evaporate.

Concretely: the label for a step expectation reads step X is Success taken from the expected status, not from whatever the step actually did. Likewise GET /path at least 1x comes from the expectation’s min, not the observed request count.

There is a regression test pinning this invariant:

a_case_label_does_not_move_when_its_assertion_flips_to_failing   (src/e2e/verify.rs)

Anyone adding a new assertion type must keep that test meaningful — build the case string from the expectation struct only.

The false-green trap

A test set that never reaches the verify engine produces zero assertions. Two ways that happens:

  1. it was skipped via skip: in case.yaml;
  2. it failed earlier — while running its flow — and never got to verification.

If those emitted nothing, the JUnit report would be green by omission: a suite where every set died before verifying would upload a file containing no failures.

set_cases in src/commands/e2e.rs handles exactly this — when a set yields no assertion-derived cases, it emits one fallback case carrying the set’s own verdict (Pass/Fail/Skip):

/// A set that never reached the verify engine — skipped, or failed while
/// running the flow — has no assertions to report, and still needs a case of
/// its own so the report does not read as green.

Note the asymmetry that is correct: sections simply absent from case.yaml are never evaluated and emit nothing at all. Absence of an expectation is not a missing test; absence of a whole set’s execution is.

Classnames

CommandClassname
mando e2e run<test set>.<section> — e.g. test_set_1.mock
mando verifybare <section> — no test set exists to scope with

Superseded by the flow-scope change ( fabc589)

mando e2e run classnames now derive from the case’s flow: field, not its directory: data-update.mock. The directory name survives only as a collision-disambiguation fallback when two cases target the same flow. Consequence, verified across pipelines 2735716225 and 2736559170: renaming a case directory (test_set_1 → data-update-beskar-soc) moves no (classname, name) pair and orphans no test history. See mando-cli-flow-coverage-2026-08-06.

mando verify gained --junit in this same change (src/cli.rs), so a standalone expectation file can now feed a CI report without going through the suite runner.

Terminal output and exit codes are unchanged by this commit.

Known gap — the always-green mock assertion

Deliberately NOT fixed; flagged for a future change

A mock.requests entry in case.yaml with none of count / min / max asserts nothing and always passes. The gron counts section guards against this shape; mock does not.

This is pre-existing behaviour, but per-assertion reporting makes it worse in appearance: the vacuous expectation now surfaces in the GitLab Tests tab as a named, green testcase labelled ... any number of times. It looks like real coverage in the MR widget while asserting nothing.

Fix direction when it is scheduled: reject a mock.requests entry with no count constraint at suite validation time, matching how gron counts already behave.

Graph View

Backlinks