Levandor

fk-dev deploy + smoke runbook

12 min read

Practical, general-purpose runbook for connecting to the fk-dev Tailscale VM, swapping any vuer component onto a branch for a smoke test (bind-mount, no image rebuild), running migrations, restarting via supervisord, and opening OpenHours so a videochat smoke actually proceeds. All facts here were live-verified on fk-dev on 2026-08-13 during the NÚSZ 1.9.11.48 smoke (branch chore/FKITDEV-9217-nusz-devel-update). Host reference: dev-build-host. NÚSZ/8959-specific variant: fk-dev-nusz-deploy-and-8959-verification. Partner-swap companion (file ownership, the read-only config/local.json, per-partner DB, operator users, css asset build): fk-dev-partner-branch-deploy-runbook.

For Agents

  • Box: fk-dev — GCP dev-mirror VM on tailnet taild4189d.ts.net, IP 100.91.108.61, MagicDNS fk-dev.taild4189d.ts.net. Stack = vuer_docker docker-compose; app source bind-mounted at /workspace/<repo>; supervisord inside each container.
  • SSH user is ops — the ONLY permitted user (verified 2026-08-13): command ssh ops@fk-dev.taild4189d.ts.net. Plain Tailscale SSH, no keypair, MagicDNS. Use command ssh because ssh is aliased to _kaku_wrapped_ssh (not loaded non-interactively).
  • GitHub fetch on the box needs command ssh -A + a github-capable key in the Mac’s ssh-agent (ssh-add ~/.ssh/id_ed25519; that key auths to GitHub as wowjeeez). The box has no GitHub key of its own.
  • vuer_css git ops must run as root inside the container (docker exec -u 0:0): its working tree has root-owned files and ops has no sudo.
  • Verify HTTP from the Mac, not from fk-dev (which can’t resolve its own sidecar MagicDNS names non-interactively).
  • OpenHours is load-bearing for any videochat/operator-handoff smoke — without it the handoff stalls. See OpenHours setup.

SSH access — ops is the only permitted user

Only ops is tailnet-permitted (verified 2026-08-13)

The tailnet SSH policy permits only ops for fk-dev and its sidecars. levander, lederera, facekom, ubuntu, dev, deploy, admin are all refused by tailnet policy.

Working invocation:

command ssh ops@fk-dev.taild4189d.ts.net
  • Plain Tailscale SSH — no keypair, no ~/.ssh config entry needed; auth is the tailnet identity + MagicDNS.
  • Use command ssh (not bare ssh): the user’s interactive shell aliases ssh to a _kaku_wrapped_ssh function that is not loaded in a non-interactive shell, so bare ssh fails with _kaku_wrapped_ssh: command not found.
  • _kaku_wrapped_ssh is a red herring for the username. It injects no user/identity — it only sets TERM and adds IdentitiesOnly. Do not read a username out of it.

Reconcile with dev-build-host — root works too (2026-08-18)

A tip block in dev-build-host (dated 2026-08-12) records root as also working, and the 2026-08-18 partner deploy connected as root successfully (fk-dev-partner-branch-deploy-runbook). The 2026-08-13 smoke below simply did not test root. Current reading: ops and root both work, everything else is refused by tailnet policy.

Which one you pick is not cosmetic — it decides who owns the files you create. Host ops is uid 1001 while the in-container app user techteamer is uid 1000, and getting that wrong sends the app into an EACCES … logs/server.log restart loop. See The four traps.

Stack layout

PropertyValue
Orchestrationvuer_docker docker-compose on fk-dev
App sourcebind-mounted per component at /workspace/vuer_oss, /workspace/vuer_css, etc.
Process managersupervisord inside each container
Containersvuer_oss, vuer_css, postgresql, janus, rabbitmq, …
Node / Yarnlive inside the containers — node 24.12, yarn 1.22.22, running as root (uid 0). The host has no node/yarn.

Because source is bind-mounted and each container runs supervisord, deploying a branch is a git checkout + in-container yarn install + supervisorctl restart — no image build, no compose recreate.

GitHub fetch on the box needs agent forwarding

The box has no GitHub key of its own

ops’s ~/.ssh is empty and the git remote is git@github.com:TechTeamer/... (SSH). Fetching therefore requires:

  1. A github-capable key loaded in the Mac’s ssh-agent: ssh-add ~/.ssh/id_ed25519 (this key authenticates to GitHub as wowjeeez).
  2. Connecting with agent forwarding: command ssh -A ops@fk-dev.taild4189d.ts.net.

With an empty agent, fetch fails with Permission denied (publickey).

Deploy recipe — switch a component onto a branch for a smoke

# On the Mac: load the github key, connect with agent forwarding
ssh-add ~/.ssh/id_ed25519
command ssh -A ops@fk-dev.taild4189d.ts.net

1. Fetch with an explicit refspec (as ops, over -A)

git -C /workspace/<repo> fetch origin '+refs/heads/<branch>:refs/remotes/origin/<branch>'

Always pass the explicit refspec

facekom clones use narrowed fetch refspecs — a plain git fetch origin <branch> writes only FETCH_HEAD and leaves origin/<branch> stale (exit 0, success message). Use the explicit +refs/heads/<branch>:refs/remotes/origin/<branch> form. (Same failure mode as narrowed-fetch-refspec-stale-devel-merge.)

2. Reset + checkout

For most components (source-owned tree):

git -C /workspace/<repo> reset --hard
git -C /workspace/<repo> clean -fd
git -C /workspace/<repo> checkout -B <branch> origin/<branch>

vuer_css has root-owned files → do git ops inside the container as root

The vuer_css working tree contains root-owned files (the container writes as root). ops has no sudo, so git clean / git checkout fail with Permission denied. Run the git ops as root inside the container:

docker exec -u 0:0 vuer_css sh -lc 'cd /workspace/vuer_css && \
  git config --global --add safe.directory /workspace/vuer_css && \
  git reset --hard && git clean -fd && \
  git checkout -B <branch> origin/<branch>'

3. Dependencies (inside the container, as root)

docker exec -u 0:0 <container> sh -lc 'cd /workspace/<repo> && yarn install --frozen-lockfile'

Run it detached and poll a logfile for long installs:

docker exec -d -u 0:0 <container> sh -lc 'cd /workspace/<repo> && yarn install --frozen-lockfile > /tmp/yarn.log 2>&1'
# then poll: docker exec <container> sh -lc 'tail -5 /tmp/yarn.log'

4. Migrations (vuer_oss)

docker exec -u 0:0 vuer_oss sh -lc 'cd /workspace/vuer_oss && npx sequelize-cli db:migrate'
  • NODE_ENV=dev; .sequelizerc → .sequelize-config.js; migration files in db/migrate.
  • The app also auto-migrates on boot, so an explicit run is usually belt-and-braces.

5. Restart via supervisord

docker exec <container> supervisorctl restart all        # or specific programs

Program lists:

  • vuer_oss: nginx, redis, vuer_oss, vuer_background, vuer_cron, vuer_integration_log, vuer_media, vuer_oss_convert, vuer_oss_storage
  • vuer_css: nginx, redis, vuer_css

6. Verify HTTP from the Mac (MagicDNS)

Curl from the Mac, not from fk-dev

fk-dev cannot resolve its own sidecar MagicDNS names in a non-interactive shell. Run the HTTP checks from the Mac:

curl -sk https://oss-fk-dev.taild4189d.ts.net    # expect 302 (login redirect)
curl -sk https://css-fk-dev.taild4189d.ts.net    # expect 200

OpenHours setup

Load-bearing for any videochat smoke

Without an open OpenHours window, the operator handoff stalls and the videochat smoke never proceeds. Open the window before starting.

  • Table: openhourstandards (plural). Model: server/db/model/openhourstandard.js.
  • Columns: dayName, from ('HH:MM', '24:00' is allowed), to, isOpen (bool), calendarName (null = default calendar).
  • The e2e helper test/tests/open-hours.setup.ts simply sets 00:00–24:00.
  • Seed reference: test/seed/seed.js getOpenHours(calendarName=null) inserts mon–sun 00:00–24:00, isOpen for the default calendar.

Open the default calendar fully

Run a Node script inside the oss container using the app’s own connection — require('./server/db/sequelize.js') reads DB creds from node-config, so no raw credentials are needed (and printing db.password gets blocked by the safety classifier anyway):

update openhourstandards
   set "from" = '00:00', "to" = '24:00', "isOpen" = true
 where "calendarName" is null;

App caches open-hours

vuer_oss may cache the open-hours; after updating the rows, clear it with:

docker exec vuer_oss supervisorctl restart vuer_oss

Janus / media server fix (verified 2026-08-13)

Symptom

Browser videochat shows “media server connection errored” and the camera never renders. Root cause: the janus container was in supervisord FATAL state — dead since the box’s first boot (2026-06-30). Its janus_websockets transport could not create the Secure WebSocket (wss) vhost: Error creating vhost for Secure Websockets server. This failed even after creating the missing certs /workspace/cert/dev.{crt,key} → the in-image libwebsockets was built without working TLS (janus built with only --enable-websockets, no LWS_WITH_TLS guarantee). Janus ports: wss 8989 / ws 8188 (signaling), wss 7989 / ws 7188 (admin).

Key architecture fact — this dissolves the problem

The browser NEVER connects to Janus. Signaling is server-to-server only: browser → vuer_css via Socket.IO(wss) → RabbitMQ → vuer_oss → Janus over a PLAIN WebSocket (@techteamer/janus-api, isomorphic-ws). The media itself flows browser ↔ operator via TURN/coturn (turnserver.facekomtest.net, config webrtc.turn.secret) — not through the Janus WS. Therefore there is NO browser mixed-content constraint and NO need for wss on the oss→Janus hop. Plain ws://localhost:8188 is correct and intended for dev.

The fix — two edits + restart

1. Enable plain ws in the janus transport config.

janus.transport.websockets.jcfg is a BIND MOUNT — edit IN-PLACE inside the container

A host-side sed -i changes the inode, and the container keeps reading the old file (wasted a cycle). Edit it inside the container so the inode is preserved:

docker exec -u 0:0 janus sh -lc 'f=/usr/local/etc/janus/janus.transport.websockets.jcfg; \
  tmp=$(sed "s/ws = \"no\"/ws = \"yes\"/; s/wss = \"yes\"/wss = \"no\"/; s/admin_wss = \"yes\"/admin_wss = \"no\"/" "$f"); \
  printf "%s\n" "$tmp" > "$f"'
docker exec janus supervisorctl start janus

Verify: Websockets server started (port 8188) appears in /var/log/janus.log.

2. Repoint vuer_oss config. NODE_ENV=dev on fk-dev, so the active file is vuer_oss/config/dev.json, key webrtc.janusServers.janus (NOT top-level janusServers — that key doesn’t exist, which is why config.get('janusServers') returns null).

  • url: wss://localhost:8989 → ws://localhost:8188
  • adminUrl: wss://localhost:7989 → ws://localhost:7188
  • keep adminSecret: "janusoverlord" — it must match admin_secret in janus.jcfg.

vuer_oss is network_mode: host → localhost is correct

Use localhost, NOT the container name. janus:8188 is ENOTFOUND — there is no container DNS on the host network.

Then restart:

docker exec vuer_oss supervisorctl restart vuer_oss

Verify

  • From oss, a WS handshake to ws://localhost:8188 with subprotocol janus-protocol sending {janus:"info"} returns server_info version=0.12.4.
  • No new Error connecting to the Janus WebSockets server in /var/log/vuer_oss.log after the restart.

Remaining dependency — TURN/coturn

Actual media needs TURN/coturn turnserver.facekomtest.net reachable from the test browser. Signaling being green does not guarantee media if TURN is unreachable.

Videochat smoke went GREEN on fk-dev 2026-08-13 — first confirmed green video on this box

The NÚSZ 1.9.11.48 videochat smoke completed end-to-end: identification flow finished “Sikeres”, and both operator and customer video rendered. Video was the untested frontier on fk-dev — now confirmed working. Record-only; do not re-verify live. Host: facekom-build-host-fk-dev.

This box had never had a confirmed green videochat before (now it has)

Video was the untested frontier on fk-dev; it is now signaling-fixed AND end-to-end green (see success callout above). Seeded operator accounts: admin / operator / supervisor @vuer.test (passwords are bcrypt-hashed; resettable via bcryptjs + a direct users-table update — see Videochat smoke gotchas (verified 2026-08-13)).

Videochat smoke gotchas (verified 2026-08-13)

Three gotchas discovered while getting the live videochat smoke green on fk-dev, after the Janus signaling fix above was already in place. These are the difference between “signaling connects” and “a real recorded call completes.”

1. Janus recording directory permission — mkdir error: 13 (Permission denied)

Symptom: signaling works, but the call won't record

With Janus signaling green, a live call still fails to record. /var/log/janus.log shows:

mkdir (/workspace/records/<roomId>/) error: 13 (Permission denied)

The janus process runs as a non-root user (techtea… = techteamer), but /workspace/records was root:root 0755 — so the process can’t create per-room subdirs. Without this, recording (and therefore room-export/conversion evidence) is broken even though the call itself may still connect and render video.

Fix — make the record root writable by the janus process user:

docker exec -u 0:0 janus sh -lc 'mkdir -p /workspace/records && chmod -R 0777 /workspace/records'
  • The record dir is driven by webrtc.janusConfig.recordDirectory (/workspace/records/).

2. “Taking a while to load” = slow ICE gathering, NOT a hang

The call DOES connect — it's just slow to gather candidates

Janus logs:

[WARN] Waiting for candidates-done callback... (slow gathering, are you using STUN
  or TURN for Janus too, instead of just for users? Consider enabling full-trickle instead)

This is not a failure. The DTLS handshake has been completed follows a few seconds later, and the operator/customer video renders after the delay. Do not chase this as a hang.

Optional speedup (not required for a green smoke): enable full_trickle = true and/or drop Janus’s own STUN / nat_1_1 in janus.jcfg — Janus doesn’t need STUN for itself on this box.

TURN reachability check — ECONNRESET on :3478 is NORMAL

Media flows over TURN (turnserver.facekomtest.net:3478); DTLS completed here, so media is fine. A raw TCP connect to :3478 returns ECONNRESET — that is expected (TURN isn’t plain TCP). Port-open is the signal, not a clean TCP session.

3. Resetting oss operator passwords (dev)

Login mechanics on this box

  • Seeded accounts: admin / operator / supervisor (emails @vuer.test, isEnabled=true).
  • Login is by USERNAME: WebServerAuth.js → User.findOne({where:{username}}), POST /login.
  • Password is sent plaintext (no client-side hashing); no 2FA / webAuthn / totp on this box.
  • The app’s bcrypt worker (workers/bcrypt) is plain bcryptjs with NO pepper / pre-hash, so a direct bcrypt.hash(pw, 10) verifies.

Reset — run a Node script inside the vuer_oss container using the app’s own connection (require('bcryptjs') + require('./server/db/sequelize.js') — no raw DB creds needed):

update users
   set "password" = '<bcryptjs-hash>',
       "passwordExpiry" = '<future-date>'
 where username in ('admin','operator');

Gotchas

  • The users table uses isEnabled (NOT isActive) and passwordExpiry — login is rejected if passwordExpiry < now, so always push it into the future.
  • A successful login is a 302 → / with a vuersid cookie; a failure redirects back to /login.
  • Browser autofill of a stale saved password is a common “password broken” false alarm — verify with curl before assuming a bad hash.

Verify end-to-end from the Mac:

curl -sk -X POST https://oss-fk-dev.taild4189d.ts.net/login \
  --data-urlencode username=admin \
  --data-urlencode password=<pw>
# expect: 302 → https://…/   (a redirect back to /login = wrong password)

Context — NÚSZ 1.9.11.48 smoke (2026-08-13)

  • Purpose: smoke the NÚSZ 1.9.11.48 release — branch chore/FKITDEV-9217-nusz-devel-update (vuer_oss d09274cb30 / vuer_css 828846cc0).
  • fk-dev was found mid-FKITDEV-9059 (cofidis) with uncommitted WIP (MJML edits + the FKITDEV-9200 k6 harness) which was overwritten per user instruction.
  • Release context: NÚSZ 1.9.11.48 test runbook.

Graph View

Backlinks