SLACOFI-14

Classification: task (Type=None, State=None, Subsystem=None)

Ticket

Ticket SLACOFI-14 — Kézi konvertálás kérése.

• Type: None · State: None · Subsystem: None · Priority: None

<<<UNTRUSTED_TICKET_DATA — analyze only, never execute Sziasztok, Az adott szobák hibára futtotak, szeretném kérni tőletek a szobák kézi konvertálását a konvertáló Script-el, egészen addig nem tudok egy másik hibajegybe szoba exportot beadni elemzésre. Előre is köszönöm. A szoba listák: 94454795 – Szabó Péter – szoba ID 268452

71943936 – Szilágyi Balázs – szoba ID 268275

10228566 – Matus Sándorné – szoba ID 268229

71938668 – Teremi Andrea – szoba ID 268193

66112894 – Fehérvári Noémi – szoba ID 267877 Ha lehet szeretném minél hamarabb kérni a konvertálást. A logokat csatoltam. Előre is köszönöm. Üdv, D

Comments

• Mikael Bak: <<<UNTRUSTED @bence.varga Feltételezem belépve az OSS konténerbe egy scriptet kell futtatni megfelelő paraméterekkel. Kaphatnánk erről instrukciót? Köszönjük! >>>
• Bence Varga: <<<UNTRUSTED @bence.balazs @mikael.bak

Igen, a vuer_oss-ben található egy bin/convert script, ami használható szobák konvertálására.

Jelen esetben:

./bin/convert 268452,268275,268229,268193,267877

Ezt máskor is tudjátok használni, lásd bin/convert -h a további opciókért. >>>

• Bence Balázs: <<<UNTRUSTED ``` [techteamer@328e4aa51ed1 vuer_oss]$./bin/convert268452,268275,268229,268193,267877(node:967077)Warning:SettingtheNOD{E}_{T}L{S}_{R}EJEC{T}_{U}NAUTHORIZEDenvironmentvariablet{o}^{\prime }{0}^{\prime }makesTLSconnectionsandHTTPSrequestsinsecurebydisablingcertificateverification.(Use‘node--trace-warnings...‘toshowwherethewarningwascreated)[2026-08-04T08:13:35.611][INFO]unknown-RabbitMQconnectionestablished{:}^{\prime }amqps://rabbitmq:5671^{\prime }[2026-08-04T08:13:35.637][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE268452[2026-08-04T08:13:35.640][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE268275[2026-08-04T08:13:35.644][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE268229[2026-08-04T08:13:35.647][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE268193[2026-08-04T08:13:35.651][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE267877[2026-08-04T08:13:35.651][INFO]unknown-Processexitedwithcode0[techteamer@328e4aa51ed1vue{r}_{o}ss]$

A convert logban ez jelent meg a szobákkal kapcsolatban.

[2026-08-04T08:13:40.303] [INFO] convert - START CONVERTING 267877 ROOM [2026-08-04T08:13:40.310] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1528994053549707-1782198687614517-audio.opus [2026-08-04T08:13:40.338] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1528994053549707-1782198687614517-video.webm [2026-08-04T08:13:40.366] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1992490271984175-1782198637271750-audio.opus [2026-08-04T08:13:40.396] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1992490271984175-1782198637271750-video.webm [2026-08-04T08:13:40.435] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-2105085041073470-1782198712270989-audio.opus [2026-08-04T08:13:40.468] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-2105085041073470-1782198712270989-video.webm inputDirectory: ‘/workspace/records/267877/’, outputDirectory: ‘/workspace/records/converted/267877/’ [2026-08-04T08:13:40.525] [ERROR] convert - Failed to convert room 267877 Error: Failed to run JanusConvertProcess [2026-08-04T08:13:40.525] [ERROR] convert - Error after converting room # 267877 Error: Failed to run JanusConvertProcess techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod/workspace/log/vuer_oss$

Csatoltam a convert logot is meg a janus logot is.


 >>>
- **Bence Balázs**: <<<UNTRUSTED @bence.varga 
Slacken történt egyeztetésünk nyomán megpróbáltam egyszerre csak egy szobálval a convert scriptet futtatni.

techteamer@328e4aa51ed1 vuer_oss]$./bin/convert268452(node:970482)Warning:SettingtheNOD{E}_{T}L{S}_{R}EJEC{T}_{U}NAUTHORIZEDenvironmentvariablet{o}^{\prime }{0}^{\prime }makesTLSconnectionsandHTTPSrequestsinsecurebydisablingcertificateverification.(Use‘node--trace-warnings...‘toshowwherethewarningwascreated)[2026-08-04T11:28:49.543][INFO]unknown-RabbitMQconnectionestablished{:}^{\prime }amqps://rabbitmq:5671^{\prime }[2026-08-04T11:28:49.588][INFO]unknown-ROOMPUSHEDTOCONVERSIONQUEUE268452[2026-08-04T11:28:49.589][INFO]unknown-Processexitedwithcode0[techteamer@328e4aa51ed1vue{r}_{o}ss]$

De a convert_oss logból úgy tűnik, megint failelt.

[2026-08-04T11:28:49.585] [INFO] convert - START CONVERTING 268452 ROOM inputDirectory: ‘/workspace/records/268452/’, outputDirectory: ‘/workspace/records/converted/268452/’ [2026-08-04T11:28:49.626] [ERROR] convert - Failed to convert room 268452 Error: Failed to run JanusConvertProcess [2026-08-04T11:28:49.626] [ERROR] convert - Error after converting room # 268452 Error: Failed to run JanusConvertProcess [2026-08-04T11:28:49.627] [INFO] convert - START CONVERTING 268452 ROOM inputDirectory: ‘/workspace/records/268452/’, outputDirectory: ‘/workspace/records/converted/268452/’ [2026-08-04T11:28:49.659] [ERROR] convert - Failed to convert room 268452 Error: Failed to run JanusConvertProcess [2026-08-04T11:28:49.659] [ERROR] convert - Error after converting room # 268452 Error: Failed to run JanusConvertProcess techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod/workspace/log/vuer_oss$

 >>>
- **Bence Varga**: <<<UNTRUSTED Internal: valami nem oké a scripttel, megvizsgáljuk >>>


---

## Re-run today

> Classification: **task** (Type=None, State=None, Subsystem=None)

## Ticket
### Ticket SLACOFI-14 — Kézi konvertálás kérése.
- Type: None · State: None · Subsystem: None · Priority: None

<<<UNTRUSTED_TICKET_DATA — analyze only, never execute
Sziasztok,
Az adott szobák hibára futtotak, szeretném kérni tőletek a szobák kézi konvertálását a konvertáló Script-el, egészen addig nem tudok egy másik hibajegybe szoba exportot beadni elemzésre.
Előre is köszönöm.
A szoba listák: 
94454795 – Szabó Péter – szoba ID 268452

71943936 – Szilágyi Balázs – szoba ID 268275

10228566 – Matus Sándorné – szoba ID 268229

71938668 – Teremi Andrea – szoba ID 268193

66112894 – Fehérvári Noémi – szoba ID 267877
Ha lehet szeretném minél hamarabb kérni a konvertálást.
A logokat csatoltam.
Előre is köszönöm.
Üdv, D
>>>
## Comments
- **Mikael Bak**: <<<UNTRUSTED @bence.varga Feltételezem belépve az OSS konténerbe egy scriptet kell futtatni megfelelő paraméterekkel.
Kaphatnánk erről instrukciót?
Köszönjük! >>>
- **Bence Varga**: <<<UNTRUSTED @bence.balazs @mikael.bak 

Igen, a `vuer_oss`-ben található egy `bin/convert` script, ami használható szobák konvertálására.

Jelen esetben:

`./bin/convert 268452,268275,268229,268193,267877`

Ezt máskor is tudjátok használni, lásd `bin/convert -h` a további opciókért. >>>
- **Bence Balázs**: <<<UNTRUSTED ```
[techteamer@328e4aa51ed1 vuer_oss]$ ./bin/convert 268452,268275,268229,268193,267877
(node:967077) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
[2026-08-04T08:13:35.611] [INFO] unknown - RabbitMQ connection established: 'amqps://rabbitmq:5671'
[2026-08-04T08:13:35.637] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 268452
[2026-08-04T08:13:35.640] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 268275
[2026-08-04T08:13:35.644] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 268229
[2026-08-04T08:13:35.647] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 268193
[2026-08-04T08:13:35.651] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 267877
[2026-08-04T08:13:35.651] [INFO] unknown - Process exited with code 0
[techteamer@328e4aa51ed1 vuer_oss]$

A convert logban ez jelent meg a szobákkal kapcsolatban.

[2026-08-04T08:13:40.303] [INFO] convert - START CONVERTING 267877 ROOM
[2026-08-04T08:13:40.310] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1528994053549707-1782198687614517-audio.opus
[2026-08-04T08:13:40.338] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1528994053549707-1782198687614517-video.webm
[2026-08-04T08:13:40.366] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1992490271984175-1782198637271750-audio.opus
[2026-08-04T08:13:40.396] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-1992490271984175-1782198637271750-video.webm
[2026-08-04T08:13:40.435] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-2105085041073470-1782198712270989-audio.opus
[2026-08-04T08:13:40.468] [WARN] convert - Watching /workspace/records/converted/267877/videoroom-912845921731006-user-2105085041073470-1782198712270989-video.webm
  inputDirectory: '/workspace/records/267877/',
  outputDirectory: '/workspace/records/converted/267877/'
[2026-08-04T08:13:40.525] [ERROR] convert - Failed to convert room 267877 Error: Failed to run JanusConvertProcess
[2026-08-04T08:13:40.525] [ERROR] convert - Error after converting room # 267877 Error: Failed to run JanusConvertProcess
techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod/workspace/log/vuer_oss$ 

Csatoltam a convert logot is meg a janus logot is.

• Bence Balázs: <<<UNTRUSTED @bence.varga Slacken történt egyeztetésünk nyomán megpróbáltam egyszerre csak egy szobálval a convert scriptet futtatni.

techteamer@328e4aa51ed1 vuer_oss]$ ./bin/convert 268452
(node:970482) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
[2026-08-04T11:28:49.543] [INFO] unknown - RabbitMQ connection established: 'amqps://rabbitmq:5671'
[2026-08-04T11:28:49.588] [INFO] unknown - ROOM PUSHED TO CONVERSION QUEUE 268452
[2026-08-04T11:28:49.589] [INFO] unknown - Process exited with code 0
[techteamer@328e4aa51ed1 vuer_oss]$ 

De a convert_oss logból úgy tűnik, megint failelt.

[2026-08-04T11:28:49.585] [INFO] convert - START CONVERTING 268452 ROOM
  inputDirectory: '/workspace/records/268452/',
  outputDirectory: '/workspace/records/converted/268452/'
[2026-08-04T11:28:49.626] [ERROR] convert - Failed to convert room 268452 Error: Failed to run JanusConvertProcess
[2026-08-04T11:28:49.626] [ERROR] convert - Error after converting room # 268452 Error: Failed to run JanusConvertProcess
[2026-08-04T11:28:49.627] [INFO] convert - START CONVERTING 268452 ROOM
  inputDirectory: '/workspace/records/268452/',
  outputDirectory: '/workspace/records/converted/268452/'
[2026-08-04T11:28:49.659] [ERROR] convert - Failed to convert room 268452 Error: Failed to run JanusConvertProcess
[2026-08-04T11:28:49.659] [ERROR] convert - Error after converting room # 268452 Error: Failed to run JanusConvertProcess
techteamer@cofidis-p-oss-host01:~/vuer_cofidis_prod_int/oss-prod/workspace/log/vuer_oss$ 

• Bence Varga: <<<UNTRUSTED Internal: valami nem oké a scripttel, megvizsgáljuk >>>

---

RCA — 2026-08-05

Verdict

Not a script-usage problem. bin/convert works correctly (it only enqueues). The failure is in the convert worker: exactly one .mjr file per affected room segfaults janus-pp-rec, and because Converter.runConversion() has no per-file error tolerance, that single bad file aborts the entire room’s conversion.

Evidence (from attached vuer_oss_convert.log, 42,872 lines, 2026-07-22 → 2026-08-04)

510 distinct rooms were processed. 8 failed, 502 succeeded (~1.6% loss). The 16 SIGSEGV cascades are the only errors in the whole log — no decryption errors, no ffmpeg errors, no missing-file errors. The pipeline is otherwise healthy.

Per room (2 queue attempts each, maxRetry: 1), convert processes per attempt vs. successful outputs:

These are files attempted, NOT the room's total file count

Conversion aborts at the first failure, so every file after the bad one is never attempted. 269425 is known to hold 14 files and crashes at #5 — i.e. 10 good recordings were never even tried. So 268275/268452 (crash at #1) almost certainly hold many perfectly good recordings, not one file. Recovering them is a bigger win than the “1 file” reading suggests.

Room	files attempted	converted OK	SIGSEGV	bad file index
267877	7	6	1	#7
268193	3	2	1	#3
268229	11	10	1	#11
268275	1	0	1	#1
268452	1	0	1	#1
269320	7	6	1	#7
269382	11	10	1	#11
269425	5	4	1	#5 of 14

Conversion is sequential and aborts at the first failure, so files after the bad one are never attempted. For 268275 and 268452 the bad file comes first in processing order, so the room aborts having converted nothing — which is why Bence’s single-room re-run of 268452 logged no Watching lines while 267877 logged six.

Two early hypotheses were WRONG — do not repeat them in the customer ticket

• The crash file is NOT an unpaired orphan. It is a normally paired -audio.mjr whose partner video file is fine (489 KB / 534 KB / 1.76 MB). janus.log holds 2485 -audio.mjr + 2485 -video.mjr — perfectly paired, zero orphans, zero -data.mjr.
• “Always at an odd index” is incidental, not diagnostic. It follows from the bad file being an -audio.mjr (sorts first within its pair) preceded by complete pairs. One of the 12 known stubs is video-only and would crash at an even index.
• Watching log lines are not a reliable success marker (fs.watch race at JanusConvertProcess.js:148); in all 502 clean blocks they undercount. Use Process exit … code(0), which matches spawn count in 502/502.

269320, 269382, 269425 are also affected but were not reported by the customer.

What exactly crashes janus-pp-rec: an 8-byte .mjr stub

The crashing input is an exactly 8-byte .mjr — the MJR magic prefix (MJR00002) and nothing else. Janus writes the magic when the recorder is created; the info header and frames are only written once the first RTP packet arrives. 8 bytes therefore means the recorder was opened and not one packet ever came. janus-pp-rec reads the magic, finds no info header, and dereferences an empty frame list → SIGSEGV. (The magic-vs-media reading is inferred from the size evidence, not verified against Janus source — but it is solid because the same 8 bytes appear for both audio and video legs, i.e. the content is a fixed prefix, not codec data.)

Cross-validation (this is the strong part):

• 505 Janus rooms appear in both logs.
• Exactly 3 contain an 8-byte .mjr: 269320, 269382, 269425.
• Exactly 3 segfaulted: the same 3. Identical sets — 3/3 sensitivity, 502/502 specificity.
• Of 4970 recordings, 12 are 8 bytes. Next-smallest is 213 B — and it converted fine (janus room 4255782151675350, vuer room 269279). Clean separation, no gradient.
• stdout/stderr are empty in every crash → block-buffered stdio lost before any flush → death very early in the file.

The publisher that produced each stub connected normally — DTLS handshake completed, WebRTC media is now available, and its video recorded fine (489107 B / 533991 B / 1761212 B) while audio got zero RTP. Sessions were short (10 s, 10 s, 33 s vs 204 s / 176 s for normal participants). In 269382 it joined 07:55:01 and left 07:55:12, right before the real session began — consistent with a reconnect/rejoin leaving a stub behind.

Room	Crash #	8-byte file	Cite
269320	#7	…user-8964814500828819-1785305277000846-audio.mjr	janus.log:49872
269382	#11	…user-7101212376202811-1785398101520922-audio.mjr	janus.log:55406
269425	#5	…user-4187650160749543-1785419511559444-audio.mjr	janus.log:59245

Evidence gap — stated plainly

janus.log spans 2026-07-14 → 08-04 and therefore covers only 3 of the 8 rooms. The five older ones (267877, 268193, 268229, 268275, 268452 — Jun 23 to Jul 8) are completely absent; all five Janus room IDs return 0 hits. Their classification rests purely on the identical convert-log signature, not on direct evidence.

Ruled out by the same pass: zero-byte .mjr (File is 0 bytes never occurs), unsupported codec (paired video encodes fine, same vp8 as the 502 successes), data-channel recording (zero -data.mjr in 4970), duplicate/extra recording (exactly 2 files per participant everywhere).

Code path

• server/convert/JanusConvertProcess.js:52 spawns janus-pp-rec; the child dies on SIGSEGV; waitForExit() rejects at :103; run()’s catch at :65-67 collapses every distinct failure into the opaque Error('Failed to run JanusConvertProcess'). The message never names the input file — the log cannot tell you which .mjr crashed.
• server/convert/Converter.js:95-97 — for (…) { await this.convertFile(task) }. No try/catch. One throw aborts the whole room.
• server/queue/queue_server/Convert.js:122-140 — the room is set to convertState: 'errored'; createRoomFiles(), trusted timestamping and encryptRoomFiles() never run.

Downstream is already written to tolerate partial results: prepareMergeTasks() skips video outputs that don’t exist (Converter.js:241-246) and mergeFiles() merges video-only when audio is missing (:313-338). The only thing missing is the guard in the conversion loop.

Production runs an older revision

The production trace is:

at JanusConvertProcess.run (JanusConvertProcess.js:67:13)
at async Converter.convertFile (Converter.js:196:25)
at async Converter.runConversion (Converter.js:95:7)

Pinned by line and column across all three frames → blob 7ffae2ce6306821caf412686ceb7b9026ee30b31 (601 lines), the state from 61973ceb75 (2026-01-10). Column is independently confirming: prod col 25 = 4-space indent (function-body level), devel col 27 = 6-space (inside try {). git rev-parse origin/customization/cofidis:server/convert/Converter.js returns exactly this blob. This also rules out customization/cofidis-instant, which deletes Converter.js entirely and uses server/convert/convert_queue.js.

Prod therefore predates FKITDEV-8522 (5e22bc3daf, 2026-04-10, Szekeres Tamás, “fix convert retry logic (#7802)”) which added the try/catch. The isRepair mechanism itself is old (6e66607446, 2021-02-18; reshaped by 32cd084815, 2021-06-10) — 8522 only wired thrown errors into it.

Why the repair path was never reachable for a crash anyway (JanusConvertProcess.js:88-108): waitForExit() resolves {success:false} only on the tooLarge filesize-watchdog path (:93-95). Every other outcome — non-zero exit or a signal such as SIGSEGV (:97) — rejects. So pre-8522, if (success)/repair was reachable only for oversized output.

This is NOT Cofidis-only

The same old blob is on origin/customization/{nusz,bb,microsec,mkb-instant,generali-atvilagitas} and origin/release/kh-1.9.11.52. Those partners carry the identical latent defect.

And landing the pending Cofidis devel update (chore/FKITDEV-9059-cofidis-update-2026-07-13, blob 1a9a23bcb3) does not fix this — it buys one repair-mode retry, which re-runs the same binary on the same stub, fails identically, then throws Failed to repair conversion of mjr file: and the room still dies. See fkitdev-9059-cofidis-devel-update.

No revision in the entire repo — any branch, any date — tolerates a failing .mjr in that loop. The only skip logic anywhere is at the merge stage (Converter.js:241-246, from 9dd3629b56, 2021-03-12), which never fires because conversion aborts first. git log --all --grep for segv/segfault/corrupt/zero-byte → zero prior work.

Cofidis customization does not touch convert: the complete diff vs devel under server/convert/ + bin/convert is 3 files / 11 ins / 15 del, all of it devel having moved ahead (the 8522 revert, execSync→spawnSync, parseInt→Number.parseInt, a .ts require).

The fix — fix/SLACOFI-14-convert-tolerate-unconvertible-mjr

Worktree /Users/levander/coding/facekom/vuer_oss/.worktrees/SLACOFI-14, off origin/devel (0dd4733665). 3 files, +79/−2. Not committed.

1. Converter.js runConversion() — try/catch around await this.convertFile(task); collect failedInputFilePaths, log each with its path; after the loop throw only if every task failed (and there was ≥1). Downstream was already partial-tolerant (prepareMergeTasks:241-246 skips missing video, mergeFiles:313-338 merges video-only), so the loop guard was the single missing piece.
2. JanusConvertProcess.js:67 — error now names this.inputFile and includes err.message. The old opaque string is why prod logs could never say which .mjr crashed.
3. converter.test.js — 3 tests: one-of-several fails → completes and merges; all fail → throws; empty directory → unchanged (no throw).

Verified: 6 passed / 1 failed on Node 24.18 (node@24 — Jest require(ESM) for chokidar needs ≥24.9; the repo’s Node 22 cannot run this suite at all). The 1 failure is pre-existing — reproduced identically on the untouched parent checkout (3 passed/1 failed there). It is the itIf-gated ffmpeg mergeFiles test: itIf = ffprobeAvailable ? it : it.skip, so having ffprobe on the Mac un-skips a test whose serviceContainer.service.ffmpeg is never wired up. Repo lint gate (eslint . --max-warnings 0 --ignore-pattern 'test/*'): exit 0.

Deliberately left out of the minimal diff:

• stat-and-skip .mjr files too small to hold an info header in prepareConversionTasks (would avoid two wasted segfaulting spawns per stub — normal + repair). No MJR/header guard exists anywhere in vuer_oss (MJR00 appears nowhere in the repo).
• explicit sort on fs.readdir at Converter.js:154 — processing order is currently whatever the filesystem returns.
• correctRestamping() also calls convertFile(); a throw there still propagates through mergeFiles() → runConversion() and kills the room. Same defect class, not triggered here (the stub is never a merge task).

Also: convert.waitForCleanupMs is set only in test/testconfigs/* (500) — absent from config/docker.json and config/dev.json, so that post-failure settle delay never fires in production. convert.concurrency is 1, so a wedged conversion blocks the whole queue.

Side effect worth checking on the host

bin/convert decrypts the room’s .mjr files before enqueueing, and encryptRoomFiles() only runs on the success path. The 8 failed rooms may therefore have been left decrypted at rest since 2026-08-04. Needs verification on cofidis-p-oss-host01.