Levandor

eSign dev-box container startup failures (nginx non-root PID) — 2026-06-01

8 min read

eSign dev-box container startup failures (nginx non-root PID) — 2026-06-01

Summary

During InstaCash 2026-05-27 devel-update testing on the lederera dev box, the esign_css and esign_oss containers came up unhealthy. Root cause is an image/infra regression, not the InstaCash update code: nginx runs as non-root techteamer (uid 1000) but the baked nginx.conf writes its PID to root-owned /run, so nginx can’t start, supervisord marks it FATAL, and the container healthcheck flips unhealthy. An ephemeral sed fix unblocks the box; the durable fix must be baked into the esign images. Two other findings here are red herrings or dev-tooling-only (RedisStore, chalk). Plus: how to actually ssh to this box from Claude’s shell.

Dev box only — host has since MOVED (2026-07-01)

Observed on the old rootless-docker box reached as Facekom (= lederera@localhost via ProxyJump FKJumpBox), now decommissioned. Current host: fk-dev Tailscale VM — see dev-build-host and §5 below. These are dev-box infra/image gotchas surfaced while testing the InstaCash update — not product bugs in the update branches. See dev-box-cv-photo-processing-failures for the sibling gotchas from the same testing wave.

1. PRIMARY — esign_css & esign_oss nginx won’t start (non-root PID file)

This is an IMAGE regression, not the InstaCash update

Both containers were unhealthy purely because nginx could not start. The node app, redis, and cron supervisord programs were all RUNNING. nginx was the only failure. Nothing in the instacash-2026-05-27 branches caused this.

Symptom

  • docker ps shows esign_css and esign_oss as unhealthy.
  • nginx fails at startup with: 
    nginx: [emerg] open("/run/nginx.pid") failed (13: Permission denied)
  • supervisorctl status inside the container:
    • nginx → FATAL Exited too quickly (process log may have details)
    • node / app, redis, cron → all RUNNING

Root cause

Both containers run images:

harbor.techteamer.com/facekom-devel/esign_css:2024.4.1-20240614
harbor.techteamer.com/facekom-devel/esign_oss:2024.4.1-20240614

For Agents

Despite the 20240614 tag, these images were rebuilt ~2025-12-08 and now ship nginx 1.28. The tag is misleading — do not assume the image is old just from the date in the tag.

The chain:

  1. The image runs as non-root user techteamer (uid 1000).
  2. Baked /etc/nginx/nginx.conf line 6 has pid /run/nginx.pid;.
  3. /run is root:root drwxr-xr-x — not writable by techteamer.
  4. nginx cannot create the PID file → [emerg] open("/run/nginx.pid") failed (13: Permission denied) → nginx exits immediately.
  5. supervisord marks nginx FATAL ("Exited too quickly").
  6. The container healthcheck (see §2) sees a non-RUNNING program → container unhealthy.

Immediate (ephemeral) fix — applied 2026-06-01

Rewrite the PID path to a techteamer-writable location and restart nginx via supervisord. Run per container (esign_css, esign_oss):

docker exec -u 0 <container> sed -i 's#^pid .*nginx.pid;#pid /tmp/nginx.pid;#' /etc/nginx/nginx.conf
docker exec <container> supervisorctl restart nginx

Verified: both containers went healthy after this.

Lost on container recreate

/etc/nginx is NOT bind-mounted (only /workspace/<svc>, the certs, and /var/log are). So this sed edit lives only inside the running container’s writable layer and is wiped on docker rm / recreate (it survives a plain docker restart since that keeps the layer, but not a recreate). It’s a stopgap to unblock the box, not a fix.

Durable fix — bake into the images

The fix has to land in the esign_css and esign_oss images / Dockerfiles, because /etc/nginx is not bind-mounted from the host checkout. Pick one:

  1. Bake pid /tmp/nginx.pid; (or another techteamer-writable path, e.g. under /workspace or /var/run/techteamer) into the image’s nginx.conf. ← simplest
  2. chown techteamer /run/nginx.pid (or chmod/own /run) at build time so the default path works.
  3. Run nginx as root in the image (least preferred — reverses the non-root hardening).

Do NOT try to fix this in the host checkout

Editing /workspace/esign_css or /workspace/esign_oss on the box does nothing for nginx — the config is baked at /etc/nginx/nginx.conf inside the image. The fix belongs in the Dockerfile / image build, then a re-pull on the dev box.

2. Container healthcheck logic (why a restart = 60s of unhealthy)

The healthcheck script is /usr/local/bin/supervisor-health-check.sh. It marks the container unhealthy if either:

  • any supervisord program is not RUNNING, OR
  • any program has uptime matching 0:00:[0-5][0-9] — i.e. up < 60 seconds (an anti-flap guard against a program that keeps crash-looping just under the check window).

Docker healthcheck params: interval 60s, retries 3, start-period 60s.

For Agents — expected behaviour after any restart

Any supervisorctl restart <program> (including the nginx fix above) makes the container report unhealthy for ~60 seconds while the restarted program’s uptime climbs back over 60s, then it recovers to healthy on its own. This is the anti-flap rule firing, not a second failure. Don’t chase it — wait out the minute and re-check.

3. RED HERRING — RedisStore is not a constructor (esign_css)

Already resolved in the code — only bites a stale node_modules

This error did not occur on the dev box (the esign_css node server was RUNNING). It surfaces only when running against an old (≤ v6, factory-style) connect-redis.

The code is correct for connect-redis v9 at server/web/web-server.js:24:

const { RedisStore } = require('connect-redis');
// ...
new RedisStore({ client });

The container’s /workspace/esign_css/node_modules is host-bind-mounted and has connect-redis@9.0.0 — matching the v9 API. So there’s no mismatch on the box.

RedisStore is not a constructor is the classic failure when the installed connect-redis is the old factory-style API (const RedisStore = require('connect-redis')(session)), i.e. a stale node_modules.

General lesson for devel updates — re- yarn install after a dependency major bump

After a dependency major bump lands in a devel-update, the running container must be re-yarn installed against the updated package.json/lockfile. A baked-image install can be stale and won’t reflect the bump. Because /workspace/<svc>/node_modules is host-bind-mounted, the install state on disk is what matters — not what the image baked. If you see RedisStore is not a constructor (or similar “X is not a constructor/function” after a major bump), suspect a stale node_modules first.

4. chalk ^5 ESM-only regression (dev tooling only — yarn trans)

Test/dev tooling only — not runtime

This breaks the yarn trans translation-check script, not any server. Fixed in the local mac checkout.

esign_oss/bin/test/trans-check.js:6 did:

const chalk = require('chalk');

Under chalk v5 (ESM-only) this throws (require() of an ESM module). Fix = convert to a dynamic import('chalk') inside the existing promise chain:

  • line 6 changed from const chalk = require('chalk') to let chalk;
  • chalk is loaded in the existing .then(...): import('chalk').then(m => { chalk = m.default; /* ...rest of chain... */ })

The fix is in the LOCAL checkout, not on the box

Applied to /Users/levander/coding/facekom/esign_oss/bin/test/trans-check.js. The dev box’s /workspace/esign_oss is a separate checkout — if yarn trans needs to run on the box, the same change must be made there too.

5. SSH access to the dev box from Claude Code’s shell

OUTDATED HOST (2026-07-01) — the Facekom / FKJumpBox topology below is DEAD

ssh Facekom (lederera@localhost via ProxyJump FKJumpBox → root@lederera-447-fk-hardver) is decommissioned (offline since ~2026-06-27). The dev host is now the fk-dev Tailscale VM: command ssh ops@fk-dev.taild4189d.ts.net (100.91.108.61, Tailscale SSH, no keypair, no jump box). See dev-build-host. The topology table is kept only as a record of the old box.

STILL TRUE — in Claude's shell, use command ssh (a kaku wrapper shadows plain ssh)

The user’s interactive shell has a kaku function that aliases ssh/scp to _kaku_wrapped_ssh. That function is not loaded in a non-interactive shell, so plain ssh fails with _kaku_wrapped_ssh: command not found. Bypass with command ssh / command scp to get the raw binary.

Old SSH topology (from ~/.ssh/config) — historical, host offline:

Host aliasResolves toNotes
Facekomlederera@localhost via ProxyJump FKJumpBox, key ~/.ssh/id_ed25519Was the rootless-docker host. DEAD.
FKJumpBoxroot@lederera-447-fk-hardverBare Alpine jump hop (no docker, no lederera user). DEAD.

Working invocation (current host)

Use a login shell (bash -l) so docker is on PATH:

command ssh ops@fk-dev.taild4189d.ts.net bash -l -s <<'EOF'
docker ps
# ... docker / supervisorctl commands here ...
EOF

Containers on the dev host: esign_css, esign_oss, vuer_oss, vuer_css, postgresql, rabbitmq, janus, nginx_proxy (crash-loops on fk-dev, bypassed by the Tailscale sidecars), plus vuer_cv on the old box (fk-dev is non-GPU; CV deferred).

Quick recipe

esign_css / esign_oss showing unhealthy on the dev box?

  1. ssh in correctly: command ssh ops@fk-dev.taild4189d.ts.net bash -l -s <<'EOF' ... EOF (plain ssh is shadowed by kaku; use a login shell so docker is on PATH). (The old ssh Facekom box is decommissioned — dev-build-host.)
  2. Find the failing program: docker exec <container> supervisorctl status. If nginx is FATAL and the app/redis/cron are RUNNING, it’s the PID-permission bug.
  3. Confirm: docker logs <container> (or nginx’s supervisord log) shows open("/run/nginx.pid") failed (13: Permission denied).
  4. Unblock (ephemeral): docker exec -u 0 <container> sed -i 's#^pid .*nginx.pid;#pid /tmp/nginx.pid;#' /etc/nginx/nginx.conf then docker exec <container> supervisorctl restart nginx. Wait ~60s (healthcheck anti-flap, §2) and re-check — it’ll go healthy.
  5. Durable: bake the PID path (or /run ownership) into the esign images — /etc/nginx is not bind-mounted, so the host checkout can’t carry it (§1).
  6. If you see RedisStore is not a constructor: stale node_modules, not this bug — re-yarn install in /workspace/<svc> (§3).

Graph View

Backlinks