Root-cause analysis of ASSMKB-113 / FKITDEV-9305, where MKB (Budai Ilona, 2026-08-28) pasted a RabbitMQ error log and a compatibility-test failure in one message and asked whether the former causes the latter. They are two unrelated problems. The RabbitMQ spam is a real, chronic, previously undetected packaging defect with a real functional consequence; the compat-test failure is a TURN relay-allocation problem that provably cannot be caused by RabbitMQ.
For Agents
Every claim below was verified from source at the exact deployed tag mkb-instant-1.9.11.67 (commit 92e235bfd9, 2026-07-14, tagged by bencelaszlo, YouTrack ASSMKB-104). MKB UAT was two tags behind on the incident date (.68 = 2026-07-29, .69 = 2026-08-14). File:line citations are the value of this note — preserve them when editing. Anything not verified is listed under Open / not established.
Corrected mid-investigation — do not re-derive the old claim
An earlier pass reported that MKB still builds from legacy vuer_build. That is wrong, and the cause was a local vuer-release checkout stale at 2026-05-14. MKB has migrated to vuer-release (projects/mkb-instant/ exists on origin/master, verified at fb78901, 2026-09-01). Verify partner-pipeline claims against origin/master, never a local checkout.
Answer to the customer’s question
Does the RabbitMQ error cause the compat-test failure?
No. Two independent proofs: (1) the ICE configuration reaches the browser through a plain page render with no queue and no RPC involved; (2) the RabbitMQ noise has been present in every shipped tag since .54 on 2025-07-01, about 14 months (see Chronology), so it cannot explain a failure that appeared recently.
Is the RabbitMQ error harmless log noise?
No. It is the symptom of a missing worker process that silently zeroes six queues in monitoring, so queue-overload alerting cannot fire for them (Consequence B — confirmed). It additionally breaks mobile SDK device-integrity verification, but only if integrityCheck.*.enable is set in MKB’s runtime config — it is absent from the committed docker config, so treat that as unconfirmed (Consequence A).
Where should MKB look for the compat-test failure?
Their TURN/coturn deployment and network path. The product surfaces no classification of its own — see No built-in differential diagnosis.
Problem 1 — no queue 'integration-log' in vhost '/' every 5 s
Root-caused. The customer log shows 36 occurrences at an exact 5-second cadence inside a 3-minute window.
The chain
The prober.vuer_oss/server/diagnostic.js — runDiagnosticTick() iterates every registered queueServer / rpcServer / queueClient / rpcClient and calls ch.checkQueue(key). checkQueue is amqplib’s passive queue.declare; against a missing queue RabbitMQ raises a channel-levelnot_found and kills the channel. getChannel()’s close handler drops the cached channel, so the next tick opens a fresh one — which is why the broker log shows a new Erlang channel pid each time on the same connection (<0.3932.0>, <0.3939.0>, <0.3946.0> … all on connection <0.1167.0>, channel 2).
The cadence.config.diagnostic.rpcRoundTripIntervalMs = 5000 (vuer_oss/config/docker.json:736, vuer_oss/config/dev.json:985), started unconditionally at vuer_oss/server.js:606. Matches the observed 5 s spacing exactly.
Why vuer_oss looked healthy. The failure is swallowed — catch pushes { messageCount: 0, consumerCount: 0 } and continues. vuer_oss logs nothing; the error exists only in the broker log. This is why the app appeared fine while RabbitMQ screamed.
Why the queue does not exist.integration-log is registered as an RPC client at vuer_oss/server.js:442-444, gated on integrationLog.enabled. In @techteamer/mq v7.2.0, RPCClient.initialize() asserts only its own reply queue (mq/src/RPCClient.ts:174) and never its target queue — unlike QueueClient, which does assert its target (mq/src/QueueClient.ts:34). So the integration-log queue exists only while integrationLog.js is running.
Who runs it.integrationLog.js is supervisor program [program:vuer_integration_log] (command=node integrationLog.js).
runDiagnosticTick() --every 5s--> ch.checkQueue('integration-log')
| |
| passive queue.declare
| v
| RabbitMQ: not_found --> channel killed
| |
+-- catch: push {0,0}, continue <----+ (vuer_oss logs NOTHING)
|
next tick opens a NEW channel
The gate is confirmed, not assumed
integrationLog.enabled is true, committed in the partner branch — vuer_oss/config/docker.json:808-809 at tag mkb-instant-1.9.11.67:
"integrationLog": { "enabled": true, "debugLog": false, "fileExportDecryptionEnabled": true, … }.
It is not a runtime override, so this is reproducible from the repo alone.
Root cause — the partner supervisor overlay is missing the worker
Root cause
MKB’s partner supervisor overlay ships only 7[program:] blocks — nginx, redis, vuer_oss, vuer_media, vuer_oss_convert, vuer_cron, vuer_background. vuer_integration_log and vuer_oss_storage are absent. The canonical repo conf has 9 (vuer_integration_log at line 50, vuer_oss_storage at line 155).
The worker never starts → nothing ever declares the queue → the diagnostic tick passive-declares a nonexistent queue forever.
The mechanism is two sequenced image builds, not layer ordering
Precise mechanism
This is not “last-write-wins within one image”. It is two separate image builds, sequenced by the build script: the base image links the canonical conf into conf.d/, and the partner image — built FROM that already-sealed base — writes its own file to the same path. The COPY structurally cannot lose.
Pipeline
Base image creates the link
Partner image overwrites it
vuer-release (current)
install/configure-app.sh:16-19, invoked from base/components/vuer_oss/Dockerfile:106
projects/mkb-instant/components/vuer_oss/Dockerfile:1-2 (FROM $BASE_COMPONENT_IMAGE_TAG) → COPY at :18
partner/mkb-instant/vuer_oss/Dockerfile:6 (FROM harbor.techteamer.com/$PROJECT_NAME/vuer_oss:…) → COPY at :31
The base is built, tagged and pushed to harbor first; the partner build then starts from it. In the legacy pipeline, partner selection comes from the git tag: build.sh:365 strips the version (mkb-instant-1.9.11.67 → mkb-instant) and sources partner/mkb-instant/settings.cfg. The legacy partner Dockerfile also runs sed -i "/user=/d" /etc/supervisor/conf.d/*.conf at :25 — belt-and-braces, since the overlay already carries no user= lines.
c78e02a — legacy vuer_build/partner/mkb-instant/ supervisor conf frozen at 7 programs. Historical origin of the drift.
2023-01-19
76a06e1970 — “FKITDEV-8: RabbitMQ cluster support (#3337)” adds the ch.checkQueue(key) loop to server/diagnostic.js. Six days after the overlay froze.
2025-06-11
Tag mkb-instant-1.9.11.53 — config/docker.json still has no integrationLog key at all. No RPC client registered ⇒ no spam.
2025-07-01
Tag mkb-instant-1.9.11.54 — first shipped tag carrying "integrationLog": { "enabled": true }. This is the binding constraint — the spam starts here. Confirmed still true at .55, .60 and the deployed .67.
2026-07-08
4bf534e — “migrate mkb-instant project (#36)” carries the same stale 116-line / 7-program file into vuer-release as projects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf.
2026-07-09
939a189 — “Release: mkb-instant, version: 1”.
2026-07-14
Tag mkb-instant-1.9.11.67 — five days after that release ⇒ the image MKB UAT runs was built through vuer-release, not the legacy pipeline.
2026-08-28
Customer reports the broker spam.
Load-bearing consequence
The spam requires all three legs above, so it begins at the latest of them: the .54 tag of 2025-07-01. The RabbitMQ error has therefore been present in every shipped tag for about 14 months before the report — chronic, not new — which is independent evidence that it cannot explain a compat-test failure that appeared recently.
Note the ordering also rules out a “queue existed and was lost” theory: vuer_integration_log was already absent from MKB’s conf (2023-01-13) before the feature was switched on (2025-07-01), so the durable queue was never created in the first place — nothing was deleted.
⚠️ Two earlier drafts of this note got this number wrong — “~3.5 years”, then “~2 years 10 months”. Method note, because the error is easy to repeat: git log -S is unreliable here.customization/mkb-instant has a non-linear history with periodic devel merges, so the key’s presence oscillates across commits on different parent paths — e.g. a73e993630 (2024-01-26) has nointegrationLog at all, despite sitting between two commits that do. The only trustworthy method is to read the value at release tags, which represent actually-shipped states. Bisection over tags gives a clean .53 absent → .54 present boundary.
Positive confirmation from the customer’s own logs
MKB’s overlay sets stdout_events_enabled=true / stderr_events_enabled=true on every program, feeding a supervisor_stdout eventlistener. That is exactly what produces the INFO nginx | … / INFO vuer_oss | … prefixes seen in the OpenShift export. A running worker would appear as INFO vuer_integration_log | ….
The 887-line export contains zero such lines. The absence is evidence, not an assumption.
(The eventlistener install is cited in the legacy partner Dockerfile:30; the vuer-release equivalent line was not separately cited, but the observed prefixes prove it is active in the deployed image.)
Fleet scope
32 of 35 partner overlays omit vuer_integration_log. Only granit, mvm, vkta keep it.
Per FKITDEV-8354-mvm-supervisor-config-dedup, the vuer_oss_storage omission is a documented deliberate repo-wide convention. There is no equivalent documentation for vuer_integration_log — and MVM keeps it while dropping storage, so this reads as drift, not design.
vuer-release commit 8959c06 (branch feat/FKITDEV-8362-mbh, see FKITDEV-8362) describes the mbh conf as “that source config minus vuer_integration_log and vuer_oss_storage”.
Only partners with integrationLog.enabled = true show the symptom — everyone else has no RPC client to probe the queue.
Consequence A — a latent functional defect, gated off in MKB’s committed config
Read this before citing Consequence A — it is CONDITIONAL, not confirmed at MKB
DeviceIntegrityCheckService.checkDeviceIntegrity()returns { isValid: true } early unless integrityCheck.apple.enable (iOS) or integrityCheck.play.enable (Android) is truthy. At tag mkb-instant-1.9.11.67 an integrityCheck block exists only in config/dev.json:367 — it is absent from both config/docker.json and config/default.json. config.get() returns its defaultValue = null for missing keys rather than throwing (config.js:139-151), so the gate is falsy and checkIntegrityToken() is never reached in the docker deployment.
⇒ The sequence below is a real defect in the code, but it does not fire at MKB unless their runtime local.json sets integrityCheck.*.enable. That file lives outside the repo and was not available, so this must be checked before the claim is used with the customer. An earlier draft of this note asserted it as confirmed impact — that was an overstatement.
The only confirmed functional impact at MKB is Consequence B.
The defect itself, if the gate is enabled
vuer_oss/server/service/IntegrityCheckServices/GooglePlayIntegrityCheckService.js (and the Apple equivalent) call await serviceContainer.rpcClient.integrationLog.createLog({...})inside try AND again inside finally, both gated on save && integrationLog.enabled. DeviceIntegrityCheckService.js:38,40 passes save = true hardcoded.
RPCClient.call() publishes to the (nonexistent) queue and waits for a reply until rpcTimeoutMs = 10000 (vuer_oss/config/docker.json:688), then rejects with RPCCLIENT MESSAGE TIMEOUT integration-log.
Sequence at MKB when a device integrity check runs:
catch logs and return false — Google/Apple is never contacted
finally fires a secondcreateLog → another ~10 s stall
A throw from finally overrides the return false → the caller gets a rejection after ~20 s
Latent smell found alongside it
checkIntegrityToken’s setTimeout(() => { throw ... }, timeout) throws from a timer callback, which the surrounding try cannot catch — it becomes an uncaughtException. Its 10000 ms default also races the identical 10000 ms RPC timeout. Not the cause here, but worth its own ticket.
Consequence B — monitoring blind spot (core bug, every affected partner)
When the not_found kills the channel, runDiagnosticTick()’s loop keeps using the dead handle. Every queue iterated after the missing one therefore reports messageCount: 0, consumerCount: 0.
channelNames iterates rpcClientslast, and integration-log is registered at server.js:442, so these are silently zeroed:
background-recognition
rpc-esign:external
cronManager
rpc-xml-report
rpc-xlsx-report
rpc-transport-css
Consequences:
The QUEUE COUNT WARNING overload alert can never fire for those six.
The admin System Settings queue table is wrong.
Any further missing queue stays invisible — only the first one ever reaches the broker.
Minimal fix ≈ 2 lines
Re-fetch the channel inside the inner loop, and this.diagnosticChannels.delete(name) in the catch.
Correction to an older vault note
cli-system-check-enhancement (2026-04-15) warns that queue data “may be undefined” if no admin is on the System Settings page. At tag mkb-instant-1.9.11.67 the tick is started unconditionally at server.js:606 — which is precisely why the broker log spams continuously with no operator present. Treat the older note’s freshness caveat as superseded for this code path.
Where the fix belongs
Do NOT delete MKB's overlay — the mbh pattern does not transfer
projects/mbh/components/vuer_oss/ has no supervisor conf (deleted in #35 / d7ebb37), so mbh correctly inherits the canonical 9-program file. MKB’s overlay is not “canonical minus 2 blocks.”
A diff of canonical vs MKB’s overlay shows that for every program the overlay:
replaces redirect_stderr=true + stdout_logfile=/var/log/<prog>.log + rotation keys with stdout_events_enabled=true / stderr_events_enabled=true / stdout_logfile=NONE / stderr_logfile=NONE, and
drops user=techteamer.
That is the supervisor-stdout eventlistener wiring that routes program output to container stdout — precisely what produces the INFO nginx | … prefixes in MKB’s Kibana export. Deleting the overlay would break MKB’s OpenShift log collection.
Independently confirmed: across the 7 shared blocks, every functional directive is identical to canonical — command, directory, environment, process_name, numprocs, umask, priority, autostart, autorestart, startsecs, exitcodes, stopsignal, stopwaitsecs. The only deltas are the removed user= lines and the file-logging → container-stdout switch. That is what makes “add one block” right and “delete the overlay” wrong.
Correct minimal fix
File:vuer-release → projects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf
(not legacy vuer_build/partner/mkb-instant/…, which has been frozen since 2023 and no longer builds MKB’s image.)
Change: add a single [program:vuer_integration_log] block, written in the overlay’s own logging style — take the canonical block (canonical conf line 50), keep every functional directive unchanged, then:
dropuser=techteamer
dropredirect_stderr and the stdout_logfile / rotation keys
Add only vuer_integration_log. Per FKITDEV-8354-mvm-supervisor-config-dedup the vuer_oss_storage omission is a deliberate repo-wide convention and must be left alone.
Deployment note for the release/test plan
integrationLog.js calls createEncryption() on startup, so the worker’s first successful start creates an integration-log encryption key. Expect that one-time state change and plan for it — it is not a no-op deployment.
Problem 2 — the compatibility test (UNRELATED)
Error: Cannot connect to STUN/TURN servers in the customer’s browser log. Not caused by Problem 1.
This is vuer_oss, not vuer_css
The compat test at MKB is served by vuer_oss. The customer’s stack frame ConnectionCheck.startChecking matches vuer_oss/client/features/system-check/check-steps/connection-check/connection-check.js, where the throw lives in startChecking(). The vuer_css twin throws from checkConnection() and is therefore not the file involved — any analysis citing vuer_css paths for this symptom is reading the wrong repo.
Route: vuer_oss/server/web/routes/compat-test.endpoint.js, registered at vuer_oss/server/web/routes.js:109.
Evidence from the customer’s 887-line OpenShift export
Window 2026-08-28 12:40–12:43 UTC:
Observation
Count
nginx requests, all 2xx/3xx/101 — zero errors, zero 5xx
384
/compat-test
200
/socket.io/
8 × 101 (WebSocket upgrade OK ⇒ the socket path was healthy)
/diagnostics/monitoring
366 × 200
mentions of turn / stun / janus / media anywhere in the export
0
the only recurring error
36 × integration-log
The nginx 111 Connection refused lines the customer quoted are a red herring
They are from 11:58, during the application restart the customer themselves mention. By 12:40 the same 127.0.0.1:10081 endpoint returns 200.
Code proof it cannot be the queue
The error string itself proves the socket leg passed. In vuer_oss/client/features/system-check/check-steps/connection-check/connection-check.js, startChecking() reads:
if (this.isSocketPassed() && this.isTurnPassed()) { return }if (this.isSocketPassed() && !this.isTurnPassed()) { throw new Error('Cannot connect to STUN/TURN servers')} else { throw new Error('Connection check failed')}
A socket failure yields 'Connection check failed', not this message. So Cannot connect to STUN/TURN servers can only be reached with the socket check green and the TURN check red. (This branch was verified in vuer_oss, not read off the vuer_css twin — it corroborates the 8× 101 log evidence independently, at code level.)
The check requires an actual relay (TURN) candidate — a STUN/srflx candidate is not enough. So this is a genuine relay-allocation failure.
The ICE server list is server-rendered into the compat-test page and read client-side from the element dataset; the peer-connection config is assembled by TurnPasswordService.getPeerConnectionConfig(...) behind the vuer_oss route above. No queue and no RPC is involved in obtaining it ⇒ RabbitMQ cannot produce this error. (The line-level citation system-check.endpoint.js:23 belongs to the vuer_css twin and does not apply here.)
Because a specifically-TURN error was thrown — not a JSON parse error — the config was delivered and parsed fine.
Janus is architecturally ruled out: the browser never talks to Janus. Signaling is browser → vuer_css Socket.IO → RabbitMQ → vuer_oss → Janus; media is browser ↔ coturn. (Same architecture fact recorded in fk-dev-deploy-smoke-runbook.)
The RabbitMQ noise predates the report by about 14 months — present in every shipped tag since .54, 2025-07-01 (see Chronology).
No built-in differential diagnosis — the constants are dead
Do not tell MKB to "read the error classification" — vuer_oss cannot produce one
In vuer_oss/client/features/webrtc/iceTest.js, AUTH_FAILED = 2 (:44) and NOT_REACHABLE = 3 (:45) are defined and read (isAuthFailed():74, isUnreachable():78) — but setResultCode() is only ever called with DONE (:136, :153) and CONNECTION_TIMED_OUT (:172). There is also no onicecandidateerror handler. They are dead constants.
Consequence: “credentials rejected”, “TURN unreachable” and “no iceServers configured”all collapse into the same message. The AUTH_FAILED / NOT_REACHABLE differential exists only in the vuer_css twin, which is not what runs here.
Two further gaps in the same file:
The candidate detail is never surfaced.this.result.ice is assigned at connection-check.js:66 but is never emitted or reported anywhere in vuer_oss — unlike vuer_css, which emits a 'report' event. Nothing reaches the UI or the server.
A silent-failure gap in the pass/fail logic:
if (isTimedOut() || isAuthFailed() || isUnreachable() || !(hasRelay || hasReflex)) passed = false
else if (hasRelay) passed = true
srflx-only — STUN works but the TURN relay does not — hits neither branch, so passed stays undefined → falsy → fails silently. TIMEOUT_PERIOD = 60000 (iceTest.js:41).
How MKB must actually diagnose it
Since the product surfaces nothing, diagnosis has to happen outside it:
chrome://webrtc-internals on the failing client (inspect the gathered candidate types),
a Trickle-ICE test against the same TURN URL and credentials,
coturn’s own server-side logs.
Candidate causes (all config/network-side)
TURN secret mismatch.TurnPasswordService.generateCredentials() builds coturn REST-style time-limited credentials: username = <unixExpiry>:<name>, password = base64(HMAC-SHA1(webrtc.turn.secret, username)). coturn’s static-auth-secretmust equal vuer_oss’s webrtc.turn.secret (or the named webrtc.turn.secrets.<name>.secret). A mismatch — or clock skew past webrtc.turn.validityInSec — rejects the allocation.
Network. UDP/3478 or TLS/5349 blocked between the bank client network and coturn, or coturn down. Common inside corporate/bank networks.
Empty iceServers after filtering.TurnPasswordService.filterByJanusServer() drops every iceServer whose janusServers array does not include the selected janus server. If none matches, iceServers ends up empty → no relay candidate → the identical error, with no server-side log.
All three are indistinguishable from the client message alone — see above.
Unrecorded history
The ticket’s own wording — “a médiaszerver hiba jelentkezik (szokásos)” — implies this recurs at MKB. That history is undocumented on our side; there is no prior ticket or note describing it.
Open / not established
Which tag MKB UAT actually ran on 2026-08-28. 1.9.11.67 is taken from the UI screenshot, not from a deployment record.
The broker log shows user: 'guest' on vhost / over port 5671, while the vault documents client-certificate RabbitMQ auth (rabbitmq-communication) — unresolved discrepancy.
Which of the three TURN candidate causes applies. The product cannot narrow it; needs client-side webrtc-internals / Trickle-ICE / coturn logs from MKB.
MKB’s recurring “szokásos” media-server failure history — undocumented on our side.
No prior art anywhere. No ticket, note or session at any partner describes the integration-log queue missing, or not_found / queue.declare spam. This is the first recorded occurrence.
Resolved since the first draft
integrationLog.enabled is verified true in config/docker.json:808-809 at the deployed tag — it is no longer an open question.
Suggested dispositions
Finding
Where it belongs
Missing vuer_integration_log at MKB
vuer-release → projects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf — add one block in the overlay’s logging style. Do not delete the overlay. Plan for the one-time createEncryption() key creation.
32/35 overlays omit vuer_integration_log
vuer_build / vuer-release — decide convention vs. drift, then either document the omission or restore the program fleet-wide
runDiagnosticTick() keeps using a dead channel (Consequence B)
vuer_oss core bug — ~2-line fix, affects every partner that hits a missing queue
Device-integrity finally-throw overriding return false (Consequence A)
vuer_oss core bug — separate from the packaging fix
checkIntegrityTokensetTimeout throw + 10 s/10 s race
Latent, own ticket
iceTest.js dead result codes, unreported result.ice, srflx-only undefined
vuer_ossdiagnosability gap — port the vuer_css classification/reporting, or this recurs unanswerable at every partner
Compat-test TURN failure itself
MKB infrastructure — request webrtc-internals / Trickle-ICE / coturn logs before any code work
Tooling gotcha reconfirmed
rtk silently mangled git show | grep into a false “no match” and rewrote git show <tag>:<file> into a tag summary. All git reads had to go through rtk proxy. Consistent with rtk-mangles-curl-and-pipes and rtk-git-log-hides-merge-commits. Compounding it: a local vuer-release checkout stale by 3.5 months produced the wrong pipeline attribution corrected at the top of this note — verify partner-pipeline claims against origin/master, never a local tree (narrowed-fetch-refspec-stale-devel-merge).
Related
ASSMKB-113 — raw customer ticket (log excerpts, screenshot, browser error)
FKITDEV-9305 — the dev-side ticket this RCA answers