Ticket Investigations
Root cause analyses produced by the Debug Agent Pipeline.
Investigations
| Ticket | Client | Severity | Status | Root Cause |
|---|---|---|---|---|
| SLANUSZ-25 | NUSZ | P1 | Root Cause Identified | IP-based rate limiter + corporate NAT blocks all operators |
| FKITDEV-8521 | MKB | Minor | Fix Ready | Phantom 3h waiting time — null cache + broad LIKE + stale cache race condition |
| FKITDEV-server-session-too-low | All | P2 | Triaged | 7 processes × 100 pool max = 700 connections, no idle/acquire/evict lifecycle |
| FKITDEV-8639 | NUSZ | P2 | Triaged | Excel export missing Sum values + locale param bug + independent rounding |
| FKITDEV-8659 | Internal | Tooling | Complete | CLI scripts for extracting and decrypting room attachments |
| CLI System Check | Internal | P2 | Triaged | Wiring task — expose existing diagnostic data to CLI system-check command |
| Room 62243 Screenshot | NUSZ | P2 | Complete | Browser never responds to videochat:screenshot:remote — 100% failure rate, 91 rooms, 31 days |
| ASSICASH-71 | InstaCash | P2 | In Review | PROD vuer_css portal.url UAT misconfig (fixed in FKITSYS-9486); WebServer.js CSP-violation logger amplifies any drift |
| FKITDEV-8787 | Raiffeisen | P2-Medium | MERGED (vuer_css PR #3064, 2026-08-10) | CORRECTED 2026-08-11: Already authorized / Already has some kind of room are server-side literals in vuer_css (server/socket/events/selfservice-v2.js:30-32), not the mobile SDK — the partner’s {"data":{"error":…}} was createEndpoint’s own reportWireClientResponse wire log. Stale socket selfServiceRoomData across in-app Restart is the real trigger; fix = fail-closed self-heal. Ships in raiffeisen-1.9.11.100; merge-back to customization/raiffeisen/devel owed. Only the ALREADY_HAS_ROOM half is fixed — auth is preserved by design |
| FKITDEV-8827 | Raiffeisen PION | Task | MERGED (vuer_oss PR #7971, 2026-08-10) | Face-comparison export for ASSRAFIPI-119 (raiffeisen-facecomparison-export.js + FaceComparisonExportService); approved by m3szi, base retargeted off the stale .95 release branch onto .100. E2E-proven against a real Postgres: a different_face row exports with status=success, confirming the read-time-verdict model. Ships in raiffeisen-1.9.11.100; merge-back owed |
| FKITDEV-8252 | Internal (fleet) | P2-Proactive | In Progress | UBI10 migration ahead of RHEL 9 EOL; vuer_build Phase A on feature/FKITDEV-8252-ubi10 (15 commits ahead, 4 Dockerfiles uncommitted); decisions on per-key rpmkeys SHA1, OL9 instantclient on UBI10, vuer_cv 2h spike; probe-builds blocked on podman sandbox |
| FKITDEV-8533 | Generali | P2 | Changes Requested | PR #7893 videoOrientExt tablet fix unreliable — modern iPadOS 13+ Safari reports a macOS UA, ua-parser-js v1 yields device.type === undefined; isTablet() is a logical subset of isMobile() (redundant). Correct fix = client-side navigator.maxTouchPoints detection |
| Face Comparison DB Query | P | P3 | Triaged (2 claims corrected 2026-08-28) | Customer premise (“difference values not in DB”) false — faceComparisons stores euclideanDistance unconditionally; different_face is a read-time threshold verdict, not a stored status. Solved with one read-only SQL query, no release. ⚠ Its liveness-gating caveat and its “status='failed' rows” claim are both wrong — see face-comparison-persistence-paths |
| Face Comparison Persistence Paths | Raiffeisen / core | Reference | Current — supersedes the two rows below on gating | Which paths actually write a row (@ tag raiffeisen-1.9.11.100). One writer, 3 call sites not 4, status always 'success' ⇒ failed comparisons structurally unrepresentable (guard failures produce no row). Kills the compareFaceWith half-truth: liveness has three handlers and only liveness-check-v2 persists — compareFaceWith is inert on liveness-check and liveness-check-v1 (what Raiffeisen/myra runs), so “just set it” is a no-op. Three ways a match happens with no row (CV-internal liveness, myra _isSameFace in-memory, silent skips); imageCategory is a copy of screenshotCategory, not a setting, and myra mislabels both sides. Carries the _isSameFace fail-open (identity 0 = best score ⇒ missing data reads as a perfect match, silently) |
| SLARAFIPI-84 | Raiffeisen (reporter Bihari Péter) | Task | Round 2 verified 2026-09-08 · State=Blocked · NOTHING POSTED SINCE 2026-08-31 | The stored “final reply” was never sent — the note said it was; the thread has 4 comments ending with the customer’s 2026-09-04 refutation. Round 2 re-derived all 19 claims at tag raiffeisen-1.9.11.100 (2352f5117f) across 4 lenses + all 8 attachments; 5 of our 9 posted 2026-08-31 claims regressed (0.55 IS the boundary; 11651 was a threshold rejection, not a recognition failure). Proof the row was never written, not filtered: Face comparison ID runs 9559→10365 with zero gaps. Recovery is viable and unencrypted (the 3 sources are not in ENCRYPTED_ACTIVITIES) and the debug flags are true at the tag. Export-side fix built on fix/SLARAFIPI-84-facecomparison-export-rejected (34/34, uncommitted). Face-comparison export complaints. 2 of 3 are not export bugs: above-threshold rows were never written (395/1202 rooms have no row); liveness comparisons are not faceComparison records at all. CSV crash = reporting.enabledExportFormats missing from config/docker.json (one-line host-config fix, no code). File mode 0o600 → asked 644 (recommend 0o640, PII with -n). Script exists only on the .100 release branch/tag — lost on the next cut from devel/customization/raiffeisen. Mechanism extracted to face-comparison-persistence-paths |
| Face Comparison Model | Raiffeisen | Reference | Current (§3 call-site table corrected 2026-08-28) | Canonical FKITDEV-8827 model note: rows key by roomId (videochat) OR selfServiceRoomId (self-service); PION = videochat (zero rows under a self-service-only query); verdict DERIVED at read time; thresholds REPLACE per-room (self-service) vs global-only (videochat); videochat rows not verdict-classified at runtime; no createdAt index; latent prepareReportData bug in the mirrored report template |
| Tailscale GCP dev-box | Internal (infra) | Infra | In Progress | DuckDNS → Tailscale migration for a GCP vuer_docker dev-box mirror over tailnet taild4189d.ts.net. 2026-06-30: MagicDNS hostnames (public-DNS retired) + 8b per-service multi-tailscaled sidecar routing; VM fk-dev provisioned (tailnet 100.91.108.61). 8b overlay now BUILT + compose config-validated on branch tailscale (unpushed) — key correction: “no app URL change” was FALSE (apps switch -→. separator off tailnet DEV_DOMAIN), fixed entirely in vuer_docker via bind-mounted per-app config/local.json (no app-repo edits). Push gated on user go; HTTPS-cert toggle pending |
| FKITDEV-8279 | Internal (bb / kh / mkb-instant / otp / unicredit-srb) | Security (CVSS 9.8) | Branch complete, not pushed | Retire the @techteamer/sequelize fork (abandoned since 2023-10-13) for upstream 6.37.8 + its three patches carried in-repo via patch-package + a boot guard + 30 tests, on fix/FKITDEV-8279-sequelize-upstream-6-37-8 @ b0303af227. Turns the currently red CI Audit job green. Two analysis claims RETRACTED (the fork was never invisible to yarn audit; the 9.8 was published 2026-08-03, not two years ago). Proven on a real Oracle 23 + SQL Server 2022 in both directions; no partner smoke test ran — fk-dev SSH denied by tailnet policy |
| FKITDEV-9194 | Generali | Release prep | Branch complete, not pushed | Generali devel update for release 1.9.11.19 on chore/FKITDEV-9194-generali-update-2026-08-08 (vuer_oss df01e922ce, vuer_css e3c8996d4). Near-miss: vuer_css’s narrowed fetch refspec silently merged a 10-day-stale devel, dropping 6bdf66d16 (FKITDEV-8887 / ASSGRALI-63) — the release’s headline item. Two semantic breaks hidden by a conflict-free merge (deleted aiActHelper.ts; auth() → SocketService.getConnection). css fully green; oss audit red on the pre-existing FKITDEV-8279 sequelize CVE, which also blocks build. Release-ticket collection (8567/9119/9080) done; browsers.oldBrowserWarning config break flagged. Generalised into devel-update-and-release-flow |
| FKITDEV-9197 | CIB | Release prep | RELEASED — tagged 2026-09-03 | CIB devel update on portal_css for release 1.9.11.102 (portal’s own tag train — baseline cib-1.4.0.74): origin/devel 56a63bd0 → customization/cib bf6dfbf8, merge base f3fb6be8 (11 months of drift), merge commit b1a4bc94 on chore/FKITDEV-9197-cib-devel-update. New break class: devel’s 7a42894a “remove unused libs” dropped multer + uuid — both used by CIB customization/api/*; multer unresolvable (real runtime break), uuid resolving only transitively. Plus 9 dead eslint-disable directives killed by devel’s eslint-9 migration under --max-warnings 0. Pre-existing devel debt recorded, not fixed: portal_css jest runs zero tests, and a dead jest-formatting rule reference. Generalised into devel-dependency-removal-breaks-partner-customization + eslint9-flat-config-dead-disable-directives OUTCOME 2026-09-03: cib-1.9.11.102 tagged (annotated) in vuer_oss 12a8a9e3 + vuer_css ca60fac3; PRs 3152 squash-merged so devel is no longer an ancestor of customization/cib (squash-merge-erases-partner-devel-ancestry); portal_css #712 left open, out of the train. Hub: cib-1.9.11.102 |
| FKITDEV-8931 | MKB | P2 | PR #3153 open (approved, 7/7 green, mergeable) | Mobile backgrounding drops the socket and the client answered with window.location.reload() — fatal for DÁP login, where backgrounding is mandatory. Fix = silent auth(socketLabel) re-auth on reconnect, gated on SILENT_REAUTH_LABELS = ['default.layout']; kiosk.layout + videochat still hard-reload by design, so kiosk-heavy mkb-instant pages are the wrong test target (use mbh-services). vuer_css only. Do not cherry-pick the FKITDEV-9199 kebab fix onto the branch |
| FKITDEV-8788 | Raiffeisen PION | P2 | Triaged | MRZ felismerési hiba (HU eID back / TD1): FULL back-side image scores MRZ valid_score:100 but the warped/cropped image of the same capture scores 2; flow rejects on the crop (VuerCVOCRRecognition warp→MRZ path). SLARAFIPI-61 was closed on a CV 4.9.0 test almost certainly run on the FULL image. Hinges on checklist #2(b): run 4.9.0 MRZ on the CROPPED 191549 export image |
| Janus memory leak RCA | Raiffeisen / DÁP / Cofidis / UniCredit (fleet) | P1 | Root Cause Identified 2026-09-02 — MEASURED ON REAL CALLS 2026-09-07 | It is OUR bug, not an upstream janus bug. vuer_oss strands live janus sessions: janus-api/src/Janus.js:407-425 keepAlive() reschedules via a setTimeout closure capturing this ⇒ a “leaked” Janus is immortal AND actively pinging, so janus never times the session out (keepAliveIntervalMs: 30000 vs the 60 s session_timeout default, which is set in no FaceKom config). MEASURED: janus reaps sessions in under 5 s on transport drop (44→4), so a crash leaks nothing — only a live pinging object strands one. Top path: server/cv/VuerCVListenerSession.js:62 leaks on every invocation incl. the success path (no destroy()/close() anywhere). Secondary 8-year-old defect: videoroom rooms are NEVER destroyed (no destroy in any janus-api videoroom plugin, since 2018-01-12) — 6.0 kB/room, but findRoom() linear-scans the whole list at 483 B/room ⇒ ~9.7 MB per call setup at 20k rooms. DEBUNKED: upstream #3408 (cited since 2024) was a Lua-plugin bug, and --disable-all-plugins excludes it; the fork changes zero lines of janus C source; 1.2.4-dev→1.4.1 buys exactly one applicable leak fix. Per-call, not per-unit-time (fk-dev idle 16 d = 20 MB; DÁP RSS flat all weekend). RELEASE BLOCKER: janus is absent from COMPONENT_LIST in all TEN Raiffeisen manifests — they are still on 0.13.4 and .101 ships nothing unless janus is added. Keep separate from FKITDEV-9193. MEASURED 2026-09-07 on an isolated real-call rig (10 cycles/arm, real media, janus RSS + Admin API census per cycle) — and it CORRECTS the hypothesis: BOTH arms leak. ABANDONED: sessions_delta 10, rooms_delta 10, rss_delta_kb 10064 ⇒ ~0.8–1.0 MB/call (reproduced: 999.2 kB/cycle), state still alive long after (9 sessions, 10 rooms, 10 DB rooms stuck incall). CLEAN: sessions_delta 0 but rooms_delta 10, rss_delta_kb 6872 ⇒ ~453 kB per properly ended call — sessions/handles ARE reclaimed, the videoroom never is. This supersedes the inferred 0.5–1.5 MB. Two side-defects: an abandoned call leaves Room.status='incall' forever ⇒ operator_in_open_room ⇒ one abandoned call takes the operator out of service; and getMainRole() returning admin silently denies videoChat.receiveCall to multi-role users (fix: POST /api/role-switch). Cron question ANSWERED: AutoCloseRoomsCronJob chain verified end-to-end and IS a real mitigation for the abandoned path, but gated on roomAutoCloseHours (no default), cannot reach VuerCVListenerSession, and is createdAt-based. Harness traps: vuer-browser-e2e-real-call-gotchas |
How to Add
New investigations are created by running the debug agent pipeline on a YouTrack ticket. The pipeline:
- Decodes the complaint (Hungarian → technical signal)
- Triages with severity, affected components, debugging checklist
- Plans reproduction using room exports and dev environment
- Investigates root cause with 5-Whys evidence chain
- Documents findings here in Obsidian
See debug-agents for full pipeline documentation.