Levandor

FaceKom release testing has three tiers — and the middle one is empty

4 min read

A per-release FaceKom test runbook is not mostly browser work — which is exactly what the k6/Playwright framing assumes. Analysed 2026-08-11 against nusz-1.9.11.48-test-runbook: roughly 60% of the runbook cannot be driven by a browser at all. The work splits into three tiers, and the middle tier has no tooling whatsoever.

For Agents

Before proposing “port more tests to k6”, check which tier the item is in. Browser tooling is the third tier and the smallest slice of a real runbook. If someone says “we have 12/45 tests ported so we’re 27% done”, that ratio is measured against tier 3 only.

The three tiers

TierNeedsCoversTooling today
1 — static / hermeticnothing; any machinelint, unit, config schema, and the clean merge, broken product detectorspartial (lint/unit exist; the detectors are specced, not built)
2 — booted, non-browserthe app running, no browserflow registration at boot, RPC calls, cron CLI completion, emails actually sending, report/xlsx exports, DB assertionsNONE
3 — browserfull stack plus mediacustomer/operator UI flowsk6 — 12 of 45 files ported

Tier 2 is the gap

It is the largest share of the runbook and it has zero harness. Every tier-2 item today is a human following prose, which is why release testing does not scale and why “green CI” keeps failing to mean “shippable”.

What tier 2 looks like in a real runbook

Concrete items from nusz-1.9.11.48-test-runbook that a browser cannot drive:

  • 12 flows registering at boot (§3.1) — boot OSS, read the flow-registration logs, confirm all 12 are present and creatable. The merge kept NÚSZ’s 12 and dropped devel’s test flows; nothing in a browser proves that.
  • Every NÚSZ email actually sending (§4.5) — MJML v5 is ESM-only; a *.letter.data.js mixing top-level import with a bare require( makes Node treat the file as ESM, require is undefined, the letter type never registers and the email silently never sends, with green CI. The runbook’s own words: silence = failure. Fourteen-plus emails must each be triggered and confirmed. See mjml-v5-esm-breaks-commonjs-email-templates.
  • remove-old-video-files cron completing (§1.1) — run node ./bin/remove-old-video-files.js -f … -t …, assert the log sequence, assert the process exits (no stdin hang on ffmpeg’s overwrite prompt), and inspect the produced converted/*.webm for a stripped video track. A CLI, a log assertion and a media-file assertion — no UI anywhere.
  • The cron set registering without throwing (§4.6) — storageUploadCronJob, audioArchive, audioDelete, removeOldRoomData, customerDelete, autoCloseRooms, flowClear, roomBulkDelete, temporaryFileCleanup all boot-register and none throws on first run.

Also tier-2-shaped in the same runbook: report/xlsx export contents and UI-language correctness (§4.1), advanced-reports RPC (§4.1 P2), and PortalData field-set persistence assertions (§4.2).

What no tooling can reach, even in principle

Worth stating separately so it is never scoped into an automation ticket:

  • iOS Safari mid-call backgrounding (§4.3 P2) — lock the device to read an SMS, return, confirm audio resumes both ways. Safari-gated, iPadOS reports a desktop UA, and it is hard to verify even on a desktop by hand.
  • The runbook’s own “cannot be validated in UAT” list: real disk reclaim, the production load profile (~40–50% CPU, ~940 Mbit/s, ~1 TB backlog), reconversion backlog at scale, and the customer-key-offline deletion branch. All prod-only.

Two hard limits on tier 3

k6 cannot do file uploads at all

This makes the attachment / S3 storage-and-export flow (§4.4) unportable — and §4.4 is a P1 on the prime conflict surface (FKITDEV-7665 S3 attachment storage, BREAKING).

The docker k6 path has no camera

k6.yml passes only CI_DOMAIN and never K6_BROWSER_ARGS — so no fake camera/mic and no --disable-web-security. For a video-identification product that is close to fatal. The k6 README.md documents those flags for native runs only. See k6-e2e-harness-vuer-oss.

The payoff: building tier 2 unblocks tier 3

k6-e2e-harness-vuer-oss lists four options for the seeding blocker that strands 30 of the 45 Playwright tests (they seed fixtures by booting the real Node app in-process via test/tests/support/helper.ts, and k6 runs its own JS VM). Option (a) is “Node pre-seed step reusing helper.ts, handing k6 a fixtures JSON via open()” — the leanest of the four.

A tier-2 harness IS that pre-seed step

Anything that boots the app in order to assert against it already has helper.ts loaded and a live DB handle. Emitting a fixtures JSON is nearly free from there. One build, two problems — which argues for building tier 2 before finishing the k6 port.

Graph View

Backlinks