Levandor

vuer_oss "Unit Tests green" is weak evidence — one dead check and two flakes

6 min read

Three independent defects mean a green Unit Tests job on vuer_oss proves less than it appears: one shared suite registers no tests at all, and two others flake non-deterministically in CI. All three are on devel, so they affect every partner branch. Found while validating the CIB devel update (FKITDEV-9197); none were changed on the partner branch, deliberately.

For Agents

  • Nothing here is CIB-specific and nothing here is merge-caused. Do not classify any of it as damage from a devel update, and do not fix it on a partner branch.
  • Two of the three live in the same file, test/tests/unit/translations.test.js.
  • Fixing #1 will convert 42 currently-invisible findings into hard failures on CIB alone. It has to be a deliberate upstream change.
  • The vuer_oss twin of portal-css-jest-runs-zero-tests — same lesson, different mechanism.

1. translations.test.js registers ZERO tests — the entire check is dead code

test/tests/unit/translations.test.js:181:

it.each(testSummary.entries(), (file, error) => { if (error) throw new Error(error) })

it.each(table) returns a function, which must then be invoked with (name, fn). Here the callback is passed as each()’s second argument and the returned function is discarded. No test is ever registered.

The suite computes its findings, then throws them away

Every wrongKeys / missingLanguage result the suite works out is silently discarded — for every partner, because the file is devel’s. The suite’s only registered tests are its it.skip(...) calls, so it contributes 0 passing tests.

Verified two ways, not assumed:

  1. Directly in node — the each() return value is a function, and nothing calls it.
  2. From CI output on a passing run: 
    Test Suites: 2 skipped, 345 passed, 345 of 347 total
Tests:       147 skipped, 10 todo, 3515 passed, 3672 total

A probe replicating the suite’s own logic found 42 findings on the CIB branch that CI cannot see — mostly flow_task_name / flow_task_instructions / flow_input_option returning undefined across customization/flow/cib-*.flow.trans.js.

Fixing the it.each is a breaking change, not a cleanup

Repairing the invocation turns those 42 findings into hard failures on CIB, and an unknown number on every other partner. It must be a deliberate upstream change with the fallout budgeted — never a drive-by alongside the [[#2-the-scan-relative-path-bug-is-actively-flaky|scan() fix]] below, and never on a partner branch.

"Translations green" ≠ "translations checked"

FKITDEV-9197’s fix acaa597d83 (registering the portal infoText / helpText / placeholder arg maps) is real — it stopped the suite throwing at import, which is what made the CIB branch’s Unit Tests job runnable. It did not make the suite validate anything, because the suite validates nothing for anyone. Say this plainly wherever the fix is cited, or someone will read the green job as coverage.

2. The scan() relative-path bug is ACTIVELY FLAKY — not merely latent

Line 28 defines the base path absolutely:

const cwd = path.join(__dirname, '../../../')

That absolute cwd is used for require and for reading the map. But includedDirectories entries are handed to scan() as relative paths and reach fs.existsSync(startPath) at line 77. Suite loading therefore depends on process.cwd() at module-evaluation time.

It is firing in production CI

Same commit, same nightly “Checks for Long-lived branches” workflow (cron * 20 * * *), branch customization/mbh, 2026-08-11:

RunTimeResult
3153181083220:13passed
3153321507020:29failed — Directory not found: client/features
3153662408021:10passed

Both runners (runnerd-techteamer-node-ci-1 and -2) produce both outcomes, so this is not one bad machine. client/features is real and git-tracked (215 files). It also failed the NÚSZ devel-update PR (run 31485303651) while barion (31475164402) and generali (31489547911) passed. Does not reproduce on macOS.

Root cause of the cwd perturbation is UNPROVEN

There is no process.chdir in repo code. The only one reachable in the dependency tree is node_modules/cross-spawn/lib/util/resolveCommand.js:18, which chdirs to options.cwd before which.sync and restores in a finally — a plausible leak window, not a demonstrated cause. Recorded as a lead, not a conclusion.

History

devel bfd1311aab (sonar code smells, PR #8000) changed line 28 from process.cwd() to the __dirname form. That fixed the require half and left scan() relative. The flake therefore both predates and survives that commit.

The obvious one-liner BREAKS the suite

path.join(cwd, startPath) alone is the wrong fix — and it fails silently

scan() builds its results from startPath (path.join(startPath, entries[i])), so absolute input makes every returned translationFile absolute. Three downstream consumers key off relative paths and all three break quietly:

ConsumerWith an absolute path
require(path.join(cwd, abs))bogus doubled path
excludedFiles.includes(abs)false ⇒ exclusions stop working
argMap[abs]undefined ⇒ every arg map silently lost

The trap underneath the trap: path.join concatenates, it does not resolve a leading / in a later segment. That is path.resolve.

The minimal correct fix scans absolutely and converts the returns back to relative:

scan(path.join(cwd, startPath), ['.trans.js', 'translations.js']).map((f) => path.relative(cwd, f))

3. A second, independent flake — pdf.test.js byte comparison

server/util/pdf.test.js › printImage › place sample PNG image does a byte-for-byte PDF comparison and drifted by 3 bytes of stream length.

  • Failed on the first CI run after the CIB push.
  • Passes on the parent commit, and 3/3 locally.
  • Cannot be reached by the CIB change: translations.map.json is read by exactly one file, translations.test.js:73.
  • Re-running the job with no content change went green.

vuer_oss's "Unit Tests green" was not first-attempt

Worth recording as method rather than trivia. Two distinct flakes are now known in this suite, so a single red run is not evidence of a regression — and, symmetrically, a single green run is not evidence of its absence.

What this means for validating a devel update

Do not treat a green test job on vuer_oss as proof

One suite silently registers nothing; two others flake non-deterministically. Concretely:

  • Never cite translations.test.js as coverage. It has none. Its green is structural.
  • Re-run before calling a red run a regression — and note that a re-run turning green tells you it was a flake, not that your change is fine.
  • Classify against a base-commit worktree (Phase 2.2) and run suspect suites isolated and repeatedly, which is exactly what distinguishes these three from real breakage.

This is the same conclusion portal-css-jest-runs-zero-tests reaches for portal_css, arrived at by a different route: on these repos the test gate is weaker evidence than its green suggests, and knowing which suites carry signal is part of the job.

Graph View

Backlinks