Levandor

Mando Known Flaky Tests

4 min read

Running catalog of known-flaky tests in the Mando repo, their root cause, and the safe remedy. Discovered while shepherding CI.

For Agents

These tests fail intermittently in CI without any code defect. Prefer retrying the CI job over blocking a merge. Real fixes go in a dedicated follow-up ticket, never bundled into an unrelated MR (zero-unrelated-changes review rule).

scheduler::tests::firing::should_fire_inner_job_through_run

Wall-clock-dependent flake in mando-bess.

  • Location: mando-bess/src/scheduler.rs:327
  • Introduced by: develop commit 3a8b8f84 (“fix: check time drift and reschedule job at each run”)
  • First observed: 2026-07-15, pipeline 2678118132, MR !570

Symptom

assertion `left == right` failed: inner job must fire
  left: 2
  right: 1

Panic timestamp was 09:08:00.656, i.e. exactly a wall-clock minute boundary. The same SHA passed an hour earlier.

Root cause

The test adds a counter job with an every-minute cron to a LIVE, already-started scheduler, calls SelfHealingJob::run(), sleeps 100ms, then asserts assert_eq!(counter, 1). If the roughly 2s test window crosses second :00 of a wall-clock minute, the real running scheduler ALSO fires the job, the counter becomes 2, and the exact-equality assertion fails. Failure probability is roughly the chance of the test window spanning a :00 boundary.

Remedy applied

Retried the failed CI job on pipeline 2678118132, which passed. The test fix was deliberately NOT included in MR !570 to respect the zero-unrelated-changes review rule.

Proper fix (needs follow-up ticket)

Either:

  • do not add the counter job to the already-started scheduler, or
  • relax the assertion to assert!(counter >= 1).

Secondary latent flake

The 100ms sleep may be too short on a loaded runner, which would fail with counter == 0. Any real fix should also give the job more headroom to fire.

debug_error::tests::gate_enabled_when_var_is_true

Env-var race between two tests in the same module. Discovered 2026-08-05 during the e2e-tests rebase.

  • Location: mando-bess/src/debug_error.rs (tests module)
  • Racing pair: gate_enabled_when_var_is_true vs gate_defaults_off_when_var_unset
  • Shared mutable state: the MANDO_DEBUG_MOCK_ERROR environment variable

Symptom

Fails roughly 2 out of 3 parallel runs. Passes reliably single-threaded.

Root cause

Both tests set/unset the same process-global env var. Under cargo’s default parallel test threads, one test clears the var while the other is asserting on it. Not a defect in the code under test.

Why CI is green

The pipeline runs --test-threads=1 (see Mando CI-CD), which serialises the pair and masks the race entirely. It only shows up in local parallel runs.

Proper fix (report to the mando team)

Either pass the gate value into the unit under test as an argument instead of reading the ambient environment, or guard every set/read/clear behind a Mutex. The identical remedy was applied to the new CSV insert route’s tests on poc/e2e-tests — see Env-var race in the CSV route tests.

Bug class, not a one-off

Any test that mutates a process-global env var is a latent cross-test race in a parallel runner. When you find one, check its whole module — they come in pairs.

test_dd_conformance.py::test_init_populates_identity_and_dd_env

Cross-test race in py-mando: leaked AlgoRunner server threads re-run a non-thread-safe logger.init(). Full write-up in pymando-logger-init-race-flaky-2026-08-10.

  • Location: py-mando/test/test_dd_conformance.py; root cause spans py-mando/src/algo_runner.rs:~113 + py_mando/python/py_mando/logger.py
  • First observed: 2026-08-10, develop ab4c54c1, pipeline 2746786029

Symptom

assert lg.identity["service"] == "bess-os-algo-optimization" fails with None. Tell: Starting Algo runner service, listening on 127.0.0.1:3003/3002 log lines interleave right before the failing test.

Root cause (short)

Constructing AlgoRunner spawns a permanent Axum server thread (leaked by test_algo_runner.py); its start_server calls logger.init() with no service arg. logger.init() sets identity then _initialized several slow lines apart (includes import ddtrace.auto), so a background init(service=None) overwrites identity with None.

Remedy applied

Double-checked threading.Lock around logger.init() shipped via MR !601 (feature/BE-4067). Narrows the window by orders of magnitude but is NOT a proven 100% fix — see the dedicated note for the real fix (stop leaking the server thread / isolate the test).

Graph View

Backlinks