Running catalog of known-flaky tests in the Mando repo, their root cause, and the safe remedy. Discovered while shepherding CI.
For Agents
These tests fail intermittently in CI without any code defect. Prefer retrying the CI job over blocking a merge. Real fixes go in a dedicated follow-up ticket, never bundled into an unrelated MR (zero-unrelated-changes review rule).
scheduler::tests::firing::should_fire_inner_job_through_run
Wall-clock-dependent flake in mando-bess.
- Location:
mando-bess/src/scheduler.rs:327 - Introduced by: develop commit
3a8b8f84(“fix: check time drift and reschedule job at each run”) - First observed: 2026-07-15, pipeline
2678118132, MR !570
Symptom
assertion `left == right` failed: inner job must fire
left: 2
right: 1
Panic timestamp was 09:08:00.656, i.e. exactly a wall-clock minute boundary. The same SHA passed an hour earlier.
Root cause
The test adds a counter job with an every-minute cron to a LIVE, already-started scheduler, calls SelfHealingJob::run(), sleeps 100ms, then asserts assert_eq!(counter, 1). If the roughly 2s test window crosses second :00 of a wall-clock minute, the real running scheduler ALSO fires the job, the counter becomes 2, and the exact-equality assertion fails. Failure probability is roughly the chance of the test window spanning a :00 boundary.
Remedy applied
Retried the failed CI job on pipeline 2678118132, which passed. The test fix was deliberately NOT included in MR !570 to respect the zero-unrelated-changes review rule.
Proper fix (needs follow-up ticket)
Either:
- do not add the counter job to the already-started scheduler, or
- relax the assertion to
assert!(counter >= 1).
Secondary latent flake
The 100ms sleep may be too short on a loaded runner, which would fail with
counter == 0. Any real fix should also give the job more headroom to fire.