Levandor

fk-dev partner-branch deploy runbook (ownership, config, DB)

16 min read

How to put a specific partner branch of vuer_oss / vuer_css onto the fk-dev VM: file ownership, the read-only per-service config/local.json, the per-partner database, operator users, and the client-asset build. Companion to fk-dev-deploy-smoke-runbook — that note owns the smoke half (janus/WebRTC fix, OpenHours, password reset, videochat gotchas); this note owns the partner swap half. Every fact below was observed live on fk-dev on 2026-08-18 while deploying vuer_css fix/FKITDEV-8931-socket-test + vuer_oss customization/mkb-instant.

For Agents

Access

root and ops are both permitted (verified 2026-08-18 with root, which was used for the whole deployment):

tailscale up                      # local Tailscale must be running
command ssh -A root@fk-dev

Refused by tailnet policy with tailnet policy does not permit you to SSH as user "<u>": levander, facekom, vuer, ubuntu, dev, techteamer, admin.

SSH-user reconciliation across notes — both work

dev-build-host (2026-08-12) records both ops and root; fk-dev-deploy-smoke-runbook (2026-08-13) recorded ops only. The 2026-08-18 deploy ran entirely as root. Settled: ops and root both work, everything else is refused.

The choice is not cosmetic:

  • ops has no sudo — anything root-owned (or any chown) must go through docker exec -u 0:0.
  • root writes root-owned files, which then need fixing up per the UID trap.

The per-service tailnet names are NOT SSH hosts

oss-fk-dev (100.91.55.42), css-fk-dev (100.106.240.87), portal-fk-dev, esign-*-fk-dev, css-sdk-demo-fk-dev are Tailscale sidecar containers. Port 22 is refused there — they serve HTTPS only. All shell work happens on fk-dev itself.

GitHub from the box

fk-dev has no GitHub credentials of its own (git@github.com: Permission denied (publickey)). Forward your agent:

# on the Mac
ssh-add --apple-load-keychain          # loads id_ed25519 (authenticates to GitHub as wowjeeez)
command ssh -A root@fk-dev
 
# on the box
export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new"

An earlier session solved this differently, by scp-ing a git bundle — the reflog still shows a since-removed bundle remote (refs/remotes/bundle/FKITDEV-9059). Agent forwarding is the simpler path.

Architecture on the box

PropertyValue
Compose project/workspace/vuer_docker (overlay documented in /workspace/vuer_docker/tailscale/README.md)
Access pathone kernel-mode tailscaled sidecar per service running tailscale serve, terminating HTTPS on <prefix>-fk-dev.taild4189d.ts.net. nginx_proxy from dev.yml is NOT the access path.
App containersnetwork_mode: host
Containersvuer_oss, vuer_css, portal_css, janus, postgresql, rabbitmq, esign_oss, esign_css (+ *-ts-1 sidecars)
Imagesharbor.techteamer.com/facekom-devel
Process managersupervisord -n -c /etc/supervisor/supervisord.conf is each app container’s CMD; per-service conf at /etc/supervisor/conf.d/supervisor_vuer_<svc>_dev.conf, autorestart=true
App identityuser techteamer, NODE_ENV=dev, DEV_DOMAIN=fk-dev.taild4189d.ts.net, cwd /workspace/vuer_<svc>, command node server.js
Portsvuer_oss web 10081 / socket 10080; vuer_css web 10083 / socket 10082
Sourcebind-mounted from the host: /workspace/vuer_oss, /workspace/vuer_css (host owner ops)
Logsbind-mounted to /workspace/vuer_docker/workspace/log/<svc>
Node / Yarninside containers only — node v24.12.0, yarn 1.22.22. The host has neither.

Supervisor programs:

  • vuer_css: nginx, redis, vuer_css
  • vuer_oss: nginx, redis, vuer_background, vuer_cron, vuer_integration_log, vuer_media, vuer_oss, vuer_oss_convert, vuer_oss_storage

The read-only config mount

config/local.json is bind-mounted read-only into each app container from:

/workspace/vuer_docker/tailscale/config/vuer_<svc>-local.json

Three consequences, all of which drive Trap 3:

  1. One file per service, not per partner. There is no …-mkb-instant-local.json.
  2. It survives branch switches. git checkout cannot change it, so switching partner means rewriting it by hand on the host.
  3. It is a single-file bind mount, so docker binds the inode. Any edit that replaces the file replaces the inode, and the container keeps reading the old one.

The four traps

1. Trap — UID mismatch: host ops is 1001, container techteamer is 1000

A tree-wide chown breaks logging — the two commands are a pair, never run only the first

Host ops is uid 1001. In-container techteamer is uid 1000, which on the host maps to ubuntu. The tree belongs to ops; logs/ belongs to 1000.

After installing or building as root, run both halves, per repo:

chown -R ops:ops /workspace/vuer_oss && chown -R 1000:1000 /workspace/vuer_oss/logs
chown -R ops:ops /workspace/vuer_css && chown -R 1000:1000 /workspace/vuer_css/logs

What happens if you run only the first (observed 2026-08-18): vuer_oss goes into a supervisor restart loop —

EACCES: permission denied, open 'logs/server.log'      # log4js / streamroller
Process exited with code 0

— with the pid climbing on each retry while supervisorctl status reports RUNNING uptime 0:00:00. The second chown fixes it.

Same shape as the esign nginx-PID failure in dev-box-esign-container-startup-failures-2026-06-01 and the janus record-dir failure in fk-dev-deploy-smoke-runbook — non-root container process, root-owned path.

The untouched sibling repo is your ownership reference

If you are unsure what a path should be owned by, stat the repo you have not touched yet:

ls -ld /workspace/vuer_css/logs      # ubuntu ubuntu  ⇒ uid 1000

"Ownership moved to 1001, the uid-1000 guidance is stale" — this is FALSE, and it is an agent artefact, not doc rot

The two uids coexist; they do not contradict each other. The working tree is 1001:1002 (host ops) and the app process plus logs/ are 1000 (techteamer in-container = ubuntu on host). Both statements are true at the same time, and both are current.

Traced provenance (2026-08-18): the claim appears in no vault note — a full-vault search found every note that touches uids stating 1000 correctly. It was generated by a historian subagent that observed the tree live as 1001:1002, correctly, and then over-generalised that single observation into “this contradicts the FKITDEV-9059 note, ownership has since changed” and “the 9059-era uid-1000 guidance is stale.” It was then repeated across an agent hand-off. Acting on it cost a supervisor restart loop.

FKITDEV-9059.md:78 was right the whole time "Container user techteamer is uid 1000 = host ubuntu, not host ops (1001)." Written 2026-07-20, still accurate. It was never stale and should not be read with suspicion. release-pipeline-automation-spec says the same.

What to be on guard against is the mechanism, not the fact. An agent reporting a live ls -l and concluding “the docs are out of date” is the failure mode here — a partial observation promoted to a correction. Trust FKITDEV-9059 and this runbook equally, and treat any hand-off claiming the uid guidance is outdated as suspect until it explains both uids.

2. Trap — yarn install as uid 1001 fails, and piping to tail hides it

Run it as root in-container, and never pipe it

Running yarn install as ops (uid 1001) dies with EACCES on pre-existing node_modules paths, e.g.

EACCES: permission denied, unlink '.../csurf/node_modules/cookie/LICENSE'

Run it as root inside the container, then restore ownership (tree back to ops/1001, and re-fix logs/ to 1000 per Trap 1).

Do not pipe yarn to tail — the pipe masks yarn’s exit code and the shell reported exit 0 while yarn had errored. Redirect and echo the status:

docker exec -u 0:0 vuer_oss sh -lc \
  'cd /workspace/vuer_oss && yarn install > /tmp/yarn.log 2>&1; echo "exit=$?"'

This is the same “green wrapper masks a failed step” trap that made a cofidis smoke over-claim in FKITDEV-9059, where bin/build/build.js exited 0 after every write had hit EACCES.

3. Trap — the mounted local.json is per-service, survives checkout, and is inode-bound

It still holds the previous partner

The mounted vuer_oss-local.json was found still carrying the previous tenant’s (CIB) db.url and a 30-entry flow.flows list. Left in place on a different partner’s branch it points the app at the wrong database (see Trap 4 for why that is worse than it sounds) and registers flows that do not exist on that branch.

But you cannot simply delete the file. config/dev.json ships hosts as empty strings, and janus as wss://localhost:8989 while fk-dev’s janus actually listens on ws://localhost:8188 (admin ws://localhost:7188 — the plain-ws switch is the janus fix).

Rule: hosts and webrtc.janusServers in local.json are mandatory box infrastructure — keep them. Drop only the partner-specific keys (db.url, flow.flows, and anything else tenant-shaped).

Rewrite it with cat > only — docker binds the inode

config/local.json is a single-file bind mount. sed -i, mv, and git checkout -- all replace the inode, and the container silently goes on reading the old file — no error, no warning, config changes that appear to do nothing. This has burned two prior sessions.

Edit in place with a redirect or heredoc:

cat > /workspace/vuer_docker/tailscale/config/vuer_oss-local.json <<'JSON'
{ … }
JSON

Independently recorded in release-pipeline-automation-spec.

4. Trap — db.syncOnStart migrates whatever db.url currently points at

A restart in the wrong order silently migrates the previous partner's database

config/dev.json sets db.syncOnStart: true, and server/bootstrap/connection/db.js runs migrate → sync → migrate on every boot. So a boot with the new partner’s code checked out and the old partner’s db.url still in local.json applies the new branch’s migrations to the old partner’s database.

Supervisor’s autorestart=true means you do not even have to restart deliberately — a crash anywhere in the checkout window is enough to trigger it.

Therefore: stop the app programs, create the new DB, and rewrite local.json BEFORE the checkout. That is why Deploy recipe is ordered the way it is.

Verified clean on 2026-08-18 — no cross-migration occurred

Audited after the fact: vuer_oss_cib holds 158 migrations, 5 CIB-only, and zero mkb-only entries. Migration-count comparison is the reliable isolation check on this box (the same technique that confirmed the cofidis repoint in release-pipeline-automation-spec).

Existing automation — vuer.sh

/workspace/vuer_docker/bin/vuer.sh already automates part of this. Its checkout path runs:

git_status → server_stop → git_checkout → dev_install → server_start → dev_build

Use it as an ordering reference, not verbatim

The stop-before-checkout shape is right and is why the recipe below adopts it. But do not run it blind:

  • No explicit fetch — so it inherits the narrowed-refspec staleness of narrowed-fetch-refspec-stale-devel-merge.
  • Hardcoded, stale branch list — it offers customization/mkb, not customization/mkb-instant.
  • No db.url handling at all — it does nothing about Trap 4, which is the one that can corrupt another partner’s database.
  • It builds AFTER starting the server (server_start → dev_build), so the box serves stale assets for the length of the build. The recipe below builds first.

vuer.sh db init seeds admin and operator with password = username.

Deploy recipe

Repos on disk are ops-owned while you are root, so every git command needs an explicit safe.directory:

git -c safe.directory=/workspace/vuer_oss -C /workspace/vuer_oss <cmd>

0. Record the outgoing state

Back up to /workspace/_restore/ before touching anything — see Rollback — /workspace/_restore/.

1. Stop the app programs

Closes the autorestart window that makes Trap 4 fire:

docker exec vuer_oss supervisorctl stop all
docker exec vuer_css supervisorctl stop all

2. Create the partner database

Per-partner convention: vuer_oss_<partner>. Databases present on the box as of 2026-08-18: vuer_oss, vuer_oss_cib, vuer_oss_cofidis, vuer_oss_test (+ the new vuer_oss_mkb_instant).

docker exec -e PGPASSWORD=dev postgresql psql -h localhost -U dev -c 'create database vuer_oss_mkb_instant'

psql needs -h localhost

Without -h, peer authentication fails. With -h localhost plus PGPASSWORD=dev -U dev it connects. (The concrete workaround for the “peer-auth blocks psql -U postgres” note in fk-dev-nusz-deploy-and-8959-verification.)

No manual migration step is needed — db.syncOnStart will run the full chain on first boot in step 6. Observed end-to-end against a brand-new empty database.

3. Rewrite the partner config

Edit /workspace/vuer_docker/tailscale/config/vuer_<svc>-local.json on the host (it is read-only inside the container), with cat > / heredoc only — see the inode warning in Trap 3. Keep hosts + webrtc.janusServers; set db.url to the new partner’s database; drop the old flow.flows.

4. Fetch + checkout

git -c safe.directory=/workspace/vuer_oss -C /workspace/vuer_oss \
    fetch origin '+refs/heads/<branch>:refs/remotes/origin/<branch>'
git -c safe.directory=/workspace/vuer_oss -C /workspace/vuer_oss \
    checkout -B <branch> origin/<branch>

The explicit refspec is mandatory — facekom clones use narrowed fetch refspecs and a plain git fetch origin <branch> leaves origin/<branch> stale while exiting 0 (narrowed-fetch-refspec-stale-devel-merge).

5. Dependencies — root, in-container, logged

docker exec -u 0:0 vuer_oss sh -lc \
  'cd /workspace/vuer_oss && yarn install > /tmp/yarn.log 2>&1; echo "exit=$?"'

Then restore ownership — both halves, per Trap 1:

chown -R ops:ops /workspace/vuer_oss && chown -R 1000:1000 /workspace/vuer_oss/logs

6. Build the vuer_css client assets — before starting

docker exec -u 0:0 vuer_css sh -lc 'cd /workspace/vuer_css && yarn build'

Any client-side change is invisible until yarn build runs

yarn build = bin/build/build.js = the external + script + style + pdfjs tasks (~10s). Output lands in gitignored directories: /workspace/vuer_css/web/{js,css,branding,polyfills,libs/pdfjs/wasm}, and nginx serves them from root /workspace/vuer_css/web. Checking out a branch alone changes nothing the browser can see.

Build before start (unlike vuer.sh) so the box never serves stale assets. And per FKITDEV-9059, the build wrapper can exit 0 having written nothing if web/ ownership is wrong — so verify the served asset, not the exit code (Verification checklist).

7. Start

docker exec vuer_oss supervisorctl start all
docker exec vuer_css supervisorctl start all

db.syncOnStart now migrates the correct database, because step 3 ran before this one.

8. Operator users

docker exec -u 1000:1000 -w /workspace/vuer_oss -e NODE_ENV=dev vuer_oss \
  node bin/db/create_user <role> <username> <password> [email] [firstName] [lastName] [phone]

Roles are defined in config/roles.json: admin, supervisor, operator. (vuer.sh db init is the alternative — it seeds admin/operator with password = username.)

The CLI takes only ONE role — the house convention is all three

bin/db/create_user accepts a single role. Grant the full set with a direct update:

update users set rights = '["admin","supervisor","operator"]' where username = 'admin';

The column is rights and it holds a JSON string. There is no role and no type column — see database-schema. For resetting an existing user’s password, use the bcryptjs recipe in 3. Resetting oss operator passwords (dev).

Check whether the partner ships a password-policy listener before blaming the password

Partners can add customization/listeners/* hooks that reject logins. cofidis’s verifypassword.js pins a fixed password and rejects admin (FKITDEV-9059). mkb-instant’s verify-password-test.js is harmless — it only rejects the literal string incorrectAccordingToPasswordPolicy, so it is not a login blocker.

Verification checklist

  1. Supervisor — every program RUNNING with non-zero uptime: 
    docker exec <container> supervisorctl status

    Uptime 0:00:00 with a changing PID = restart loop, and supervisorctl says RUNNING either way RUNNING is not the signal — uptime is. A healthy service accumulates uptime; a looping one is re-spawned faster than you can read it, so uptime stays pinned at 0:00:00 while the pid changes between calls. Run supervisorctl status twice and compare the pid. On 2026-08-18 this was the EACCES … logs/server.log of Trap 1.

  2. Logs show both listeners: Web server is listening on 1008x and Socket server is listening on 1008x.
  3. HTTPS from the Mac (fk-dev cannot resolve its own sidecar MagicDNS non-interactively): 
    curl -s -o /dev/null -w "%{http_code}" https://css-fk-dev.taild4189d.ts.net    # 200
    The first hit can take ~16s while the sidecar warms up. Not a failure.
  4. Socket.IO handshake: 
    curl -s "https://<svc>-fk-dev.taild4189d.ts.net/socket.io/?EIO=4&transport=polling"
# 0{"sid":…,"upgrades":["websocket"],…}
  5. Grep the bundle as SERVED over HTTPS, not the file on disk — this is the only check that distinguishes “branch checked out” from “browser gets the new code”: 
    curl -sk https://css-fk-dev.taild4189d.ts.net/js/default/default.layout.js | grep SILENT_REAUTH_LABELS
  6. Assert the referenced assets return 200, not just the HTML.

    HTML 200 with every stylesheet 404 is a real, observed failure mode still exited 0 (FKITDEV-9059). Pull the asset URLs out of the HTML and check each one.

    A prior session had a 200 page with correct partner markers while every stylesheet 404’d, because the build had written nothing and

  7. Login: 
    curl -sk -o /dev/null -w '%{redirect_url}' -X POST https://oss-fk-dev.taild4189d.ts.net/login \
  --data-urlencode username=admin --data-urlencode password=<pw>
    Success redirects to /; failure redirects back to /login.

Two false alarms — do not re-chase these

(a) An old mtime on web/branding/layouts means nothing

The directory mtime is stale because the .branding.css files inside are overwritten in place; the files carry the real build time (verified: all Aug 18 08:57). Stat the files, not the directory.

(b) mkb-instant's verify-password-test.js is not a login blocker

It rejects only the literal string incorrectAccordingToPasswordPolicy. Contrast cofidis’s verifypassword.js, which really does pin a fixed password (FKITDEV-9059).

Rollback — /workspace/_restore/

The box keeps prior-tenant state under /workspace/_restore/ so a partner swap is reversible:

  • nusz-state-*.txt, vuer_oss-local.json.nusz.bak, and a janus-fix patch (from earlier rounds)
  • added 2026-08-18: cib-state-20260818-0852.txt, vuer_oss-local.json.cib.bak, vuer_css-local.json.cib.bak

Follow the precedent: before swapping partners, drop a <partner>-state-<YYYYMMDD-HHMM>.txt (branch + commit per repo, DB name, login) and back up both vuer_{oss,css}-local.json there. Restore the config with cat bak > file — never mv, per the inode warning in Trap 3.

Box state left by the 2026-08-18 deploy

vuer_cssfix/FKITDEV-8931-socket-test @ 3f32334ae
vuer_osscustomization/mkb-instant @ c7ceaac2da
Databasevuer_oss_mkb_instant
Loginadmin / Facekom123!
Previouslycib-9197 against vuer_oss_cib — restorable from /workspace/_restore/

Partner-branch context for customization/mkb-instant: customization-branches. Full ticket note: FKITDEV-8931.

Testing FKITDEV-8931 on this box — scope matters

The fix fires ONLY for socketLabel default.layout

kiosk.layout and videochat still hard-reload — that is out of scope for FKITDEV-8931, not a regression. mkb-instant is heavily kiosk-based, so testing on a kiosk page will look like a failure.

Correct test target: https://css-fk-dev.taild4189d.ts.net/ lands on mbh-services, which extends default.layout and is the non-kiosk DÁP path.

Reconnect timing, the patch itself, and the do-not-cherry-pick warning are in FKITDEV-8931.

Browser-tested 2026-08-18 — a long outage still reloads the page, and the threshold is random

The short-drop case passes (~1.4s → silent re-auth, no reload). But the fix only cancels a pre-existing reload countdown; it does not remove it. A 40s outage (supervisorctl stop vuer_css) hard-reloaded the page 4× against a 502. The countdown is hideDelay: 10 + random(50) ⇒ 10–60s, re-rolled on every page load, so this reproduces intermittently.

Also: connectionStateRecovery never engages here (socket.recovered === false on every reconnect) — do not test against a 30s boundary. Full results in FKITDEV-8931.

Graph View

Backlinks