Root-cause analysis of ASSMKB-113 / FKITDEV-9305, where MKB (Budai Ilona, 2026-08-28) pasted a RabbitMQ error log and a compatibility-test failure in one message and asked whether the former causes the latter. They are two unrelated problems. The RabbitMQ spam is a real, chronic, previously undetected packaging defect with a real functional consequence; the compat-test failure is a TURN relay-allocation problem that provably cannot be caused by RabbitMQ.

For Agents

Every claim below was verified from source at the exact deployed tag mkb-instant-1.9.11.67 (commit 92e235bfd9, 2026-07-14, tagged by bencelaszlo, YouTrack ASSMKB-104). MKB UAT was two tags behind on the incident date (.68 = 2026-07-29, .69 = 2026-08-14). File:line citations are the value of this note — preserve them when editing. Anything not verified is listed under Open / not established.

Corrected mid-investigation — do not re-derive the old claim

An earlier pass reported that MKB still builds from legacy vuer_build. That is wrong, and the cause was a local vuer-release checkout stale at 2026-05-14. MKB has migrated to vuer-release (projects/mkb-instant/ exists on origin/master, verified at fb78901, 2026-09-01). Verify partner-pipeline claims against origin/master, never a local checkout.

Answer to the customer’s question

Does the RabbitMQ error cause the compat-test failure?No. Two independent proofs: (1) the ICE configuration reaches the browser through a plain page render with no queue and no RPC involved; (2) the RabbitMQ noise has been present in every shipped tag since .54 on 2025-07-01, about 14 months (see Chronology), so it cannot explain a failure that appeared recently.
Is the RabbitMQ error harmless log noise?No. It is the symptom of a missing worker process that silently zeroes six queues in monitoring, so queue-overload alerting cannot fire for them (Consequence B — confirmed). It additionally breaks mobile SDK device-integrity verification, but only if integrityCheck.*.enable is set in MKB’s runtime config — it is absent from the committed docker config, so treat that as unconfirmed (Consequence A).
Where should MKB look for the compat-test failure?Their TURN/coturn deployment and network path. The product surfaces no classification of its own — see No built-in differential diagnosis.

Problem 1 — no queue 'integration-log' in vhost '/' every 5 s

Root-caused. The customer log shows 36 occurrences at an exact 5-second cadence inside a 3-minute window.

The chain

  1. The prober. vuer_oss/server/diagnostic.jsrunDiagnosticTick() iterates every registered queueServer / rpcServer / queueClient / rpcClient and calls ch.checkQueue(key). checkQueue is amqplib’s passive queue.declare; against a missing queue RabbitMQ raises a channel-level not_found and kills the channel. getChannel()’s close handler drops the cached channel, so the next tick opens a fresh one — which is why the broker log shows a new Erlang channel pid each time on the same connection (<0.3932.0>, <0.3939.0>, <0.3946.0> … all on connection <0.1167.0>, channel 2).
  2. The cadence. config.diagnostic.rpcRoundTripIntervalMs = 5000 (vuer_oss/config/docker.json:736, vuer_oss/config/dev.json:985), started unconditionally at vuer_oss/server.js:606. Matches the observed 5 s spacing exactly.
  3. Why vuer_oss looked healthy. The failure is swallowedcatch pushes { messageCount: 0, consumerCount: 0 } and continues. vuer_oss logs nothing; the error exists only in the broker log. This is why the app appeared fine while RabbitMQ screamed.
  4. Why the queue does not exist. integration-log is registered as an RPC client at vuer_oss/server.js:442-444, gated on integrationLog.enabled. In @techteamer/mq v7.2.0, RPCClient.initialize() asserts only its own reply queue (mq/src/RPCClient.ts:174) and never its target queue — unlike QueueClient, which does assert its target (mq/src/QueueClient.ts:34). So the integration-log queue exists only while integrationLog.js is running.
  5. Who runs it. integrationLog.js is supervisor program [program:vuer_integration_log] (command=node integrationLog.js).
runDiagnosticTick()  --every 5s-->  ch.checkQueue('integration-log')
        |                                    |
        |                          passive queue.declare
        |                                    v
        |                          RabbitMQ: not_found  --> channel killed
        |                                    |
        +-- catch: push {0,0}, continue <----+   (vuer_oss logs NOTHING)
                                             |
                                    next tick opens a NEW channel

The gate is confirmed, not assumed

integrationLog.enabled is true, committed in the partner branchvuer_oss/config/docker.json:808-809 at tag mkb-instant-1.9.11.67: "integrationLog": { "enabled": true, "debugLog": false, "fileExportDecryptionEnabled": true, … }. It is not a runtime override, so this is reproducible from the repo alone.

Root cause — the partner supervisor overlay is missing the worker

Root cause

MKB’s partner supervisor overlay ships only 7 [program:] blocks — nginx, redis, vuer_oss, vuer_media, vuer_oss_convert, vuer_cron, vuer_background. vuer_integration_log and vuer_oss_storage are absent. The canonical repo conf has 9 (vuer_integration_log at line 50, vuer_oss_storage at line 155).

The worker never starts → nothing ever declares the queue → the diagnostic tick passive-declares a nonexistent queue forever.

The mechanism is two sequenced image builds, not layer ordering

Precise mechanism

This is not “last-write-wins within one image”. It is two separate image builds, sequenced by the build script: the base image links the canonical conf into conf.d/, and the partner image — built FROM that already-sealed base — writes its own file to the same path. The COPY structurally cannot lose.

PipelineBase image creates the linkPartner image overwrites it
vuer-release (current)install/configure-app.sh:16-19, invoked from base/components/vuer_oss/Dockerfile:106projects/mkb-instant/components/vuer_oss/Dockerfile:1-2 (FROM $BASE_COMPONENT_IMAGE_TAG) → COPY at :18
vuer_build (legacy)base/vuer_oss/Dockerfile:241ln -s /workspace/vuer_oss/supervisor_vuer_oss_docker.conf /etc/supervisor/conf.d/partner/mkb-instant/vuer_oss/Dockerfile:6 (FROM harbor.techteamer.com/$PROJECT_NAME/vuer_oss:…) → COPY at :31

The base is built, tagged and pushed to harbor first; the partner build then starts from it. In the legacy pipeline, partner selection comes from the git tag: build.sh:365 strips the version (mkb-instant-1.9.11.67mkb-instant) and sources partner/mkb-instant/settings.cfg. The legacy partner Dockerfile also runs sed -i "/user=/d" /etc/supervisor/conf.d/*.conf at :25 — belt-and-braces, since the overlay already carries no user= lines.

Same layering family as FKITDEV-8354-mvm-supervisor-config-dedup and unversioned-partner-supervisor-overlays.

Chronology — this is chronic, not new

DateEvent
2023-01-13c78e02a — legacy vuer_build/partner/mkb-instant/ supervisor conf frozen at 7 programs. Historical origin of the drift.
2023-01-1976a06e1970“FKITDEV-8: RabbitMQ cluster support (#3337)” adds the ch.checkQueue(key) loop to server/diagnostic.js. Six days after the overlay froze.
2025-06-11Tag mkb-instant-1.9.11.53config/docker.json still has no integrationLog key at all. No RPC client registered ⇒ no spam.
2025-07-01Tag mkb-instant-1.9.11.54 — first shipped tag carrying "integrationLog": { "enabled": true }. This is the binding constraint — the spam starts here. Confirmed still true at .55, .60 and the deployed .67.
2026-07-084bf534e“migrate mkb-instant project (#36)” carries the same stale 116-line / 7-program file into vuer-release as projects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf.
2026-07-09939a189“Release: mkb-instant, version: 1”.
2026-07-14Tag mkb-instant-1.9.11.67five days after that release ⇒ the image MKB UAT runs was built through vuer-release, not the legacy pipeline.
2026-08-28Customer reports the broker spam.

Load-bearing consequence

The spam requires all three legs above, so it begins at the latest of them: the .54 tag of 2025-07-01. The RabbitMQ error has therefore been present in every shipped tag for about 14 months before the report — chronic, not new — which is independent evidence that it cannot explain a compat-test failure that appeared recently.

Note the ordering also rules out a “queue existed and was lost” theory: vuer_integration_log was already absent from MKB’s conf (2023-01-13) before the feature was switched on (2025-07-01), so the durable queue was never created in the first place — nothing was deleted.

⚠️ Two earlier drafts of this note got this number wrong — “~3.5 years”, then “~2 years 10 months”. Method note, because the error is easy to repeat: git log -S is unreliable here. customization/mkb-instant has a non-linear history with periodic devel merges, so the key’s presence oscillates across commits on different parent paths — e.g. a73e993630 (2024-01-26) has no integrationLog at all, despite sitting between two commits that do. The only trustworthy method is to read the value at release tags, which represent actually-shipped states. Bisection over tags gives a clean .53 absent → .54 present boundary.

Positive confirmation from the customer’s own logs

MKB’s overlay sets stdout_events_enabled=true / stderr_events_enabled=true on every program, feeding a supervisor_stdout eventlistener. That is exactly what produces the INFO nginx | … / INFO vuer_oss | … prefixes seen in the OpenShift export. A running worker would appear as INFO vuer_integration_log | ….

The 887-line export contains zero such lines. The absence is evidence, not an assumption.

(The eventlistener install is cited in the legacy partner Dockerfile:30; the vuer-release equivalent line was not separately cited, but the observed prefixes prove it is active in the deployed image.)

Fleet scope

  • 32 of 35 partner overlays omit vuer_integration_log. Only granit, mvm, vkta keep it.
  • Per FKITDEV-8354-mvm-supervisor-config-dedup, the vuer_oss_storage omission is a documented deliberate repo-wide convention. There is no equivalent documentation for vuer_integration_log — and MVM keeps it while dropping storage, so this reads as drift, not design.
  • vuer-release commit 8959c06 (branch feat/FKITDEV-8362-mbh, see FKITDEV-8362) describes the mbh conf as “that source config minus vuer_integration_log and vuer_oss_storage.
  • Only partners with integrationLog.enabled = true show the symptom — everyone else has no RPC client to probe the queue.

Consequence A — a latent functional defect, gated off in MKB’s committed config

Read this before citing Consequence A — it is CONDITIONAL, not confirmed at MKB

DeviceIntegrityCheckService.checkDeviceIntegrity() returns { isValid: true } early unless integrityCheck.apple.enable (iOS) or integrityCheck.play.enable (Android) is truthy. At tag mkb-instant-1.9.11.67 an integrityCheck block exists only in config/dev.json:367 — it is absent from both config/docker.json and config/default.json. config.get() returns its defaultValue = null for missing keys rather than throwing (config.js:139-151), so the gate is falsy and checkIntegrityToken() is never reached in the docker deployment.

⇒ The sequence below is a real defect in the code, but it does not fire at MKB unless their runtime local.json sets integrityCheck.*.enable. That file lives outside the repo and was not available, so this must be checked before the claim is used with the customer. An earlier draft of this note asserted it as confirmed impact — that was an overstatement.

The only confirmed functional impact at MKB is Consequence B.

The defect itself, if the gate is enabled

vuer_oss/server/service/IntegrityCheckServices/GooglePlayIntegrityCheckService.js (and the Apple equivalent) call await serviceContainer.rpcClient.integrationLog.createLog({...}) inside try AND again inside finally, both gated on save && integrationLog.enabled. DeviceIntegrityCheckService.js:38,40 passes save = true hardcoded.

RPCClient.call() publishes to the (nonexistent) queue and waits for a reply until rpcTimeoutMs = 10000 (vuer_oss/config/docker.json:688), then rejects with RPCCLIENT MESSAGE TIMEOUT integration-log.

Sequence at MKB when a device integrity check runs:

  1. trycreateLog~10 s stallRPCCLIENT MESSAGE TIMEOUT integration-log
  2. catch logs and return falseGoogle/Apple is never contacted
  3. finally fires a second createLoganother ~10 s stall
  4. A throw from finally overrides the return false → the caller gets a rejection after ~20 s

Latent smell found alongside it

checkIntegrityToken’s setTimeout(() => { throw ... }, timeout) throws from a timer callback, which the surrounding try cannot catch — it becomes an uncaughtException. Its 10000 ms default also races the identical 10000 ms RPC timeout. Not the cause here, but worth its own ticket.

Consequence B — monitoring blind spot (core bug, every affected partner)

When the not_found kills the channel, runDiagnosticTick()’s loop keeps using the dead handle. Every queue iterated after the missing one therefore reports messageCount: 0, consumerCount: 0.

channelNames iterates rpcClients last, and integration-log is registered at server.js:442, so these are silently zeroed:

  • background-recognition
  • rpc-esign:external
  • cronManager
  • rpc-xml-report
  • rpc-xlsx-report
  • rpc-transport-css

Consequences:

  1. The QUEUE COUNT WARNING overload alert can never fire for those six.
  2. The admin System Settings queue table is wrong.
  3. Any further missing queue stays invisible — only the first one ever reaches the broker.

Minimal fix ≈ 2 lines

Re-fetch the channel inside the inner loop, and this.diagnosticChannels.delete(name) in the catch.

Correction to an older vault note

cli-system-check-enhancement (2026-04-15) warns that queue data “may be undefined” if no admin is on the System Settings page. At tag mkb-instant-1.9.11.67 the tick is started unconditionally at server.js:606 — which is precisely why the broker log spams continuously with no operator present. Treat the older note’s freshness caveat as superseded for this code path.

Where the fix belongs

Do NOT delete MKB's overlay — the mbh pattern does not transfer

projects/mbh/components/vuer_oss/ has no supervisor conf (deleted in #35 / d7ebb37), so mbh correctly inherits the canonical 9-program file. MKB’s overlay is not “canonical minus 2 blocks.”

A diff of canonical vs MKB’s overlay shows that for every program the overlay:

  • replaces redirect_stderr=true + stdout_logfile=/var/log/<prog>.log + rotation keys with stdout_events_enabled=true / stderr_events_enabled=true / stdout_logfile=NONE / stderr_logfile=NONE, and
  • drops user=techteamer.

That is the supervisor-stdout eventlistener wiring that routes program output to container stdout — precisely what produces the INFO nginx | … prefixes in MKB’s Kibana export. Deleting the overlay would break MKB’s OpenShift log collection.

Independently confirmed: across the 7 shared blocks, every functional directive is identical to canonical — command, directory, environment, process_name, numprocs, umask, priority, autostart, autorestart, startsecs, exitcodes, stopsignal, stopwaitsecs. The only deltas are the removed user= lines and the file-logging → container-stdout switch. That is what makes “add one block” right and “delete the overlay” wrong.

Correct minimal fix

File: vuer-releaseprojects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf (not legacy vuer_build/partner/mkb-instant/…, which has been frozen since 2023 and no longer builds MKB’s image.)

Change: add a single [program:vuer_integration_log] block, written in the overlay’s own logging style — take the canonical block (canonical conf line 50), keep every functional directive unchanged, then:

  • drop user=techteamer
  • drop redirect_stderr and the stdout_logfile / rotation keys
  • add stdout_events_enabled=true, stderr_events_enabled=true, stdout_logfile=NONE, stderr_logfile=NONE

Add only vuer_integration_log. Per FKITDEV-8354-mvm-supervisor-config-dedup the vuer_oss_storage omission is a deliberate repo-wide convention and must be left alone.

Deployment note for the release/test plan

integrationLog.js calls createEncryption() on startup, so the worker’s first successful start creates an integration-log encryption key. Expect that one-time state change and plan for it — it is not a no-op deployment.


Problem 2 — the compatibility test (UNRELATED)

Error: Cannot connect to STUN/TURN servers in the customer’s browser log. Not caused by Problem 1.

This is vuer_oss, not vuer_css

The compat test at MKB is served by vuer_oss. The customer’s stack frame ConnectionCheck.startChecking matches vuer_oss/client/features/system-check/check-steps/connection-check/connection-check.js, where the throw lives in startChecking(). The vuer_css twin throws from checkConnection() and is therefore not the file involved — any analysis citing vuer_css paths for this symptom is reading the wrong repo.

Route: vuer_oss/server/web/routes/compat-test.endpoint.js, registered at vuer_oss/server/web/routes.js:109.

Evidence from the customer’s 887-line OpenShift export

Window 2026-08-28 12:40–12:43 UTC:

ObservationCount
nginx requests, all 2xx/3xx/101 — zero errors, zero 5xx384
/compat-test200
/socket.io/8 × 101 (WebSocket upgrade OK ⇒ the socket path was healthy)
/diagnostics/monitoring366 × 200
mentions of turn / stun / janus / media anywhere in the export0
the only recurring error36 × integration-log

The nginx 111 Connection refused lines the customer quoted are a red herring

They are from 11:58, during the application restart the customer themselves mention. By 12:40 the same 127.0.0.1:10081 endpoint returns 200.

Code proof it cannot be the queue

  • The error string itself proves the socket leg passed. In vuer_oss/client/features/system-check/check-steps/connection-check/connection-check.js, startChecking() reads:
    if (this.isSocketPassed() && this.isTurnPassed()) { return }
    if (this.isSocketPassed() && !this.isTurnPassed()) {
      throw new Error('Cannot connect to STUN/TURN servers')
    } else {
      throw new Error('Connection check failed')
    }
    A socket failure yields 'Connection check failed', not this message. So Cannot connect to STUN/TURN servers can only be reached with the socket check green and the TURN check red. (This branch was verified in vuer_oss, not read off the vuer_css twin — it corroborates the 8× 101 log evidence independently, at code level.)
  • The check requires an actual relay (TURN) candidate — a STUN/srflx candidate is not enough. So this is a genuine relay-allocation failure.
  • The ICE server list is server-rendered into the compat-test page and read client-side from the element dataset; the peer-connection config is assembled by TurnPasswordService.getPeerConnectionConfig(...) behind the vuer_oss route above. No queue and no RPC is involved in obtaining it ⇒ RabbitMQ cannot produce this error. (The line-level citation system-check.endpoint.js:23 belongs to the vuer_css twin and does not apply here.)
  • Because a specifically-TURN error was thrown — not a JSON parse error — the config was delivered and parsed fine.
  • Janus is architecturally ruled out: the browser never talks to Janus. Signaling is browser → vuer_css Socket.IO → RabbitMQ → vuer_oss → Janus; media is browser ↔ coturn. (Same architecture fact recorded in fk-dev-deploy-smoke-runbook.)
  • The RabbitMQ noise predates the report by about 14 months — present in every shipped tag since .54, 2025-07-01 (see Chronology).

No built-in differential diagnosis — the constants are dead

Do not tell MKB to "read the error classification" — vuer_oss cannot produce one

In vuer_oss/client/features/webrtc/iceTest.js, AUTH_FAILED = 2 (:44) and NOT_REACHABLE = 3 (:45) are defined and read (isAuthFailed() :74, isUnreachable() :78) — but setResultCode() is only ever called with DONE (:136, :153) and CONNECTION_TIMED_OUT (:172). There is also no onicecandidateerror handler. They are dead constants.

Consequence: “credentials rejected”, “TURN unreachable” and “no iceServers configured” all collapse into the same message. The AUTH_FAILED / NOT_REACHABLE differential exists only in the vuer_css twin, which is not what runs here.

Two further gaps in the same file:

  1. The candidate detail is never surfaced. this.result.ice is assigned at connection-check.js:66 but is never emitted or reported anywhere in vuer_oss — unlike vuer_css, which emits a 'report' event. Nothing reaches the UI or the server.
  2. A silent-failure gap in the pass/fail logic:
    if (isTimedOut() || isAuthFailed() || isUnreachable() || !(hasRelay || hasReflex)) passed = false
    else if (hasRelay)                                                                 passed = true
    
    srflx-only — STUN works but the TURN relay does not — hits neither branch, so passed stays undefined → falsy → fails silently. TIMEOUT_PERIOD = 60000 (iceTest.js:41).

How MKB must actually diagnose it

Since the product surfaces nothing, diagnosis has to happen outside it:

  • chrome://webrtc-internals on the failing client (inspect the gathered candidate types),
  • a Trickle-ICE test against the same TURN URL and credentials,
  • coturn’s own server-side logs.

Candidate causes (all config/network-side)

  1. TURN secret mismatch. TurnPasswordService.generateCredentials() builds coturn REST-style time-limited credentials: username = <unixExpiry>:<name>, password = base64(HMAC-SHA1(webrtc.turn.secret, username)). coturn’s static-auth-secret must equal vuer_oss’s webrtc.turn.secret (or the named webrtc.turn.secrets.<name>.secret). A mismatch — or clock skew past webrtc.turn.validityInSec — rejects the allocation.
  2. Network. UDP/3478 or TLS/5349 blocked between the bank client network and coturn, or coturn down. Common inside corporate/bank networks.
  3. Empty iceServers after filtering. TurnPasswordService.filterByJanusServer() drops every iceServer whose janusServers array does not include the selected janus server. If none matches, iceServers ends up empty → no relay candidate → the identical error, with no server-side log.

All three are indistinguishable from the client message alone — see above.

Unrecorded history

The ticket’s own wording — “a médiaszerver hiba jelentkezik (szokásos)” — implies this recurs at MKB. That history is undocumented on our side; there is no prior ticket or note describing it.


Open / not established

  • Which tag MKB UAT actually ran on 2026-08-28. 1.9.11.67 is taken from the UI screenshot, not from a deployment record.
  • The broker log shows user: 'guest' on vhost / over port 5671, while the vault documents client-certificate RabbitMQ auth (rabbitmq-communication) — unresolved discrepancy.
  • Which of the three TURN candidate causes applies. The product cannot narrow it; needs client-side webrtc-internals / Trickle-ICE / coturn logs from MKB.
  • MKB’s recurring “szokásos” media-server failure history — undocumented on our side.
  • No prior art anywhere. No ticket, note or session at any partner describes the integration-log queue missing, or not_found / queue.declare spam. This is the first recorded occurrence.

Resolved since the first draft

integrationLog.enabled is verified true in config/docker.json:808-809 at the deployed tag — it is no longer an open question.

Suggested dispositions

FindingWhere it belongs
Missing vuer_integration_log at MKBvuer-releaseprojects/mkb-instant/components/vuer_oss/supervisor_vuer_oss_docker.conf — add one block in the overlay’s logging style. Do not delete the overlay. Plan for the one-time createEncryption() key creation.
32/35 overlays omit vuer_integration_logvuer_build / vuer-release — decide convention vs. drift, then either document the omission or restore the program fleet-wide
runDiagnosticTick() keeps using a dead channel (Consequence B)vuer_oss core bug — ~2-line fix, affects every partner that hits a missing queue
Device-integrity finally-throw overriding return false (Consequence A)vuer_oss core bug — separate from the packaging fix
checkIntegrityToken setTimeout throw + 10 s/10 s raceLatent, own ticket
iceTest.js dead result codes, unreported result.ice, srflx-only undefinedvuer_oss diagnosability gap — port the vuer_css classification/reporting, or this recurs unanswerable at every partner
Compat-test TURN failure itselfMKB infrastructure — request webrtc-internals / Trickle-ICE / coturn logs before any code work

Tooling gotcha reconfirmed

rtk silently mangled git show | grep into a false “no match” and rewrote git show <tag>:<file> into a tag summary. All git reads had to go through rtk proxy. Consistent with rtk-mangles-curl-and-pipes and rtk-git-log-hides-merge-commits. Compounding it: a local vuer-release checkout stale by 3.5 months produced the wrong pipeline attribution corrected at the top of this note — verify partner-pipeline claims against origin/master, never a local tree (narrowed-fetch-refspec-stale-devel-merge).