Levandor

DuckDNS → Tailscale migration for a GCP vuer_docker dev-box mirror

12 min read

Mirror the FaceKom physical dev box on a GCP VM and reach it over the Tailscale tailnet taild4189d.ts.net instead of DuckDNS / a public IP. Source-verified against vuer_docker @ devel. As of 2026-06-30 the architecture is decided, the VM (fk-dev) is provisioned, and the 8b sidecar overlay is BUILT + docker compose config-validated on the local tailscale branch (unpushed) — see 2026-06-30 — 8b overlay implemented (unpushed). The original planning analysis is preserved below the resolution for context.

For Agents

Status: IN PROGRESS. Decisions landed via the babylon #facekom_dev channel on 2026-06-30: (1) hostnames use Tailscale MagicDNS (<name>.taild4189d.ts.net), NOT public facekomdev.net subdomains — this retired the Phase 2a public-DNS approach below; (2) routing = 8b (per-service multi-tailscaled sidecar), NOT port-based 8a. The deploy agent provisioned the VM fk-dev (e2-standard-4, europe-west6-a, tailnet IP 100.91.108.61, MagicDNS fk-dev.taild4189d.ts.net). The 8b overlay (tailscale.yml + serve/config/README files) is now BUILT and merge-validated on the local vuer_docker branch tailscale (no commit, unpushed) — see 2026-06-30 — 8b overlay implemented (unpushed). Key correction: the “no app URL change” assumption was FALSE — fixed entirely inside vuer_docker via bind-mounted config/local.json per app (no app-repo edits). Next action (on user go): push, then deploy wires onto fk-dev. Related vuer_docker dev-env work: FKITDEV-8252 (UBI10 migration). Update 2026-07-03: the overlay is now pushed + running on fk-dev — see 2026-07-03 — fk-dev operational: 8b overlay live + NÚSZ deployed.

2026-07-03 — fk-dev operational: 8b overlay live + NÚSZ deployed

The 8b Tailscale-sidecar overlay is no longer gated — it is pushed, wired, and running on fk-dev. Confirmed operationally: the per-service sidecars are up (oss-fk-dev, css-fk-dev, esign-*-fk-dev, portal-fk-dev), the vuer_oss operator UI serves at https://oss-fk-dev.taild4189d.ts.net (HTTP 302; Express :10081 inside), and app source is bind-mounted at /workspace/<repo> with supervisord per container. This closes the “push gated on user go” open item tracked in the sections below.

Access is command ssh ops@fk-dev.taild4189d.ts.net (Tailscale SSH, no keypair; the kaku wrapper shadows ssh in Claude’s shell → use command ssh). First real use: the two NÚSZ fixes (FKITDEV-8747 + FKITDEV-8959) were deployed via a bind-mount branch checkout and FKITDEV-8959 TC-8959-02 was verified. The deploy recipe (git agent-forwarding, in-container yarn build, supervisorctl restart all) and the verification are documented in fk-dev-nusz-deploy-and-8959-verification.

Still open

janus / WebRTC media over the tailnet remains unsolved; CV is deferred (VM is non-GPU). nginx_proxy crash-loops but is bypassed by the sidecars (harmless). The postgresql container uses peer-auth (no psql -U postgres — reach the DB via the app’s Sequelize connection).

2026-06-30 — Decisions landed + VM provisioned

Two decisions landed via the babylon #facekom_dev channel; the deploy agent then provisioned the VM.

Decision 1 — Hostname scheme: Tailscale MagicDNS

Use Tailscale MagicDNS names (<name>.taild4189d.ts.net), not facekomdev.net subdomains. This retires the public-DNS Phase 2a approach described later in this note (no public *.facekomdev.net A-record). Certs come from tailscale cert against the MagicDNS names.

Decision 2 — Routing: 8b multi-tailscaled sidecar per service

The deploy agent chose 8b (multi-tailscaled sidecar per service) over port-based 8a. Each of the 9 services gets:

  • its own userspace tailscaled sidecar (env TS_AUTHKEY + TS_HOSTNAME, ~30 MB idle),
  • its own MagicDNS name,
  • its own tailscale cert.

No app URL rework expected — IFF naming preserved

No application URL rework is expected provided sidecars are named to preserve the existing <prefix>-<DEV_DOMAIN> pattern (e.g. oss-fk-dev, css-fk-dev) with DEV_DOMAIN=fk-dev.taild4189d.ts.net. This is flagged to verify against the actual app config before committing. Pattern is proven on pmv2-zurich (babylon + nats sidecars).

VM provisioned: fk-dev

The deploy agent stood up the VM as part of a levandor-infra terraform refactor (module "vm" promoted to for_each = var.vms); pmv2-zurich’s 14 prod containers were verified untouched.

PropertyValue
Machine typee2-standard-4 (4 vCPU / 16 GB)
Disk100 GB pd-balanced
Zoneeurope-west6-a
Networkseparate VPC fk-dev-net / subnet 10.2.0.0/24
Service accountfk-dev-sa
Tailnet IP100.91.108.61
MagicDNSfk-dev.taild4189d.ts.net
ACL tagtag:cloud (reuses the pmv2 auth key)
Public IP34.158.19.122 — egress-only
Firewalldenies ALL inbound except Tailscale DERP; no public SSH
Docker29.6.1 (+ gcloud cred helper for europe-west3-docker.pkg.dev)
Othertailscaled + OTel collector active (OTel at host.docker.internal:4317/4318)
Accessssh ops@fk-dev from any tailnet device

Open items (owner: Andras/user)

Awaiting user

  1. CV / GPU scope — current VM is non-GPU.
  2. Toggle “HTTPS Certificates” ON in the Tailscale admin console — one-click, required before tailscale cert works.
  3. Whether deploy creates a FaceKom Artifact Registry namespace.

Next (this agent, on user go)

Build the 8b compose sidecar layer on the vuer_docker branch tailscale:

  • one sidecar per service named <prefix>-fk-dev,
  • tailscale serve https → localhost:<port>,
  • likely drop nginx_proxy (per-service sidecars replace the single reverse proxy),
  • verify app cross-service URLs against the preserved <prefix>-<DEV_DOMAIN> pattern,
  • then push for deploy to wire onto fk-dev.

2026-06-30 — 8b overlay implemented (unpushed)

The 8b multi-tailscaled-sidecar overlay is BUILT and docker compose config-validated on the vuer_docker branch tailscale (local, no commit, unpushed). Status stays IN PROGRESS pending the user’s push go.

Key correction — “no app URL change” was FALSE

The earlier assumption that preserving the <prefix>-<DEV_DOMAIN> naming meant zero app changes was wrong. Apps compute their hostname separator as:

separator = DEV_DOMAIN.endsWith('facekomdev.net') ? '-' : '.'

So a tailnet DEV_DOMAIN (e.g. fk-dev.taild4189d.ts.net) flips the separator to ., producing invalid dotted multi-label names like oss.fk-dev.taild4189d.ts.net — which is not resolvable by MagicDNS and not the sidecar’s name (oss-fk-dev.taild4189d.ts.net). The naming-preservation caveat flagged in the prior section is therefore not satisfiable by env-var alone.

Fix — entirely inside vuer_docker, zero app-repo edits

Two behaviours were verified empirically in the app/config layer:

  1. Every app’s host derivation is guarded if (!config.X) — an explicitly-set host short-circuits the DEV_DOMAIN-based computation.
  2. The getconfig library deep-merges config/local.json last — a test proved local.json overrides one key while preserving sibling keys.

So vuer_docker ships a config/local.json per app, bind-mounted into each container at /workspace/<app>/config/local.json, setting hosts.* explicitly to <prefix>-fk-dev.taild4189d.ts.net. No app repository is touched.

Files added on branch tailscale (unpushed, no commit)

  • tailscale.yml — 8 userspace tailscale/tailscale:stable sidecars (network_mode: host, TS_USERSPACE=true, --advertise-tags=tag:cloud, per-service TS_HOSTNAME, TS_SERVE_CONFIG), DRY via YAML anchors, named ts-state-* volumes; plus 5 app-service stubs that merge in the local.json mounts.
  • tailscale/serve/{oss,css,css-sdk-demo,esign-oss,esign-api,esign-css,portal,library}.json — each a tailscale serve HTTPS (${TS_CERT_DOMAIN}:443) → http://127.0.0.1:<port> mapping. Ports: 20080 / 30080 / 30081 / 20180 / 20181 / 30180 / 30380 / 50080.
  • tailscale/config/{vuer_oss,vuer_css,esign_oss,esign_css,portal_css}-local.json — explicit hosts.* (plus portal.url for esign); vuer_oss sets hosts.cv=null (CV out of scope).
  • tailscale/README.md; .gitignore += tailscale/tailscale.env.

Validation

docker compose -f dev.yml -f vuer-oss.yml -f vuer-css.yml -f esign-oss.yml \
  -f esign-css.yml -f portal-css.yml -f facekom-library.yml -f tailscale.yml config -q
# EXIT 0

Merge is valid; all 5 local.json mounts and 8 serve mounts are present in the merged config.

Networking model

Each sidecar is host-net + userspace tailscaled; tailscale serve proxies to 127.0.0.1:<app-port> on the host. The apps stay unchanged (still network_mode: host). Each service gets its own MagicDNS name + own tailscale cert. The nginx_proxy from dev.yml is no longer the access path (harmless if left running).

Caveats

  • Requires the “HTTPS Certificates” toggle ON in the tailnet admin console (owner: Andras) before cert issuance works.
  • vuer_oss hosts.api=api-fk-dev has no sidecar — but api- is only used by the customization /external/createCustomerToken hostname-gating, not the base dev flow (matches today’s unrouted api-).

Open / next

  • Push is gated on explicit user go. Once pushed, deploy wires it onto fk-dev (the deploy mechanism stays ours).
  • HTTPS toggle pending (admin console).
  • CV deferred (VM is non-GPU; hosts.cv=null).
  • No new Artifact Registry — uses the existing one.

2026-06-30 — HTTPS enabled + impl reviewed + ansible task

HTTPS cert issuance unblocked on the tailnet, the 8b overlay passed correctness review with no code changes, three tailnet-admin prereqs were flagged to Andras, and a dev-workflow ansible task was posted to deploy. Status stays IN PROGRESS (push still gated on Andras’s go).

HTTPS Certificates enabled

Andras toggled “HTTPS Certificates” ON in the Tailscale admin console. Per-node tailscale cert issuance now works for the sidecars — this clears the prior blocker flagged under Open items (owner: Andras/user) and the 8b caveats.

Correctness review of the kernel-mode 8b overlay — PASSED

Reviewed the built overlay end-to-end: PASSED, no code changes needed. Key insight: the new topology (tailscale serve TLS-terminate → http://app:port) is identical in shape to the old nginx_proxy, so the app side behaves unchanged. Verified:

  • docker compose merges clean.
  • Apps stay network_mode: host.
  • Apps listen on 0.0.0.0, so host.docker.internal:host-gateway reaches them (Docker 29.6.1 supports host-gateway).
  • App nginx serves HTTP on the ports — matches serve → http.
  • Each sidecar gets its own bridge netns → own tailscale0 (no conflict).
  • local.json host overrides make CORS / SAML / socket Origin pass.

Three tailnet-admin prereqs flagged (owner: Andras)

Tailnet-admin prereqs before wiring

  1. Auth key must be NON-ephemeral. The deploy’s existing tag:cloud key is reusable but EPHEMERAL — that risks a MagicDNS name flap (oss-fk-dev → oss-fk-dev-1) which would break the hardcoded hosts + cert binding. Use a reusable, non-ephemeral, pre-authorized tag:cloud key (the ts-state-* volumes already persist identity).
  2. ACL grant: team devices → tag:cloud:443 (+ MagicDNS), or they resolve but can’t connect.
  3. Runtime check at wiring: confirm tailscale serve preserves the original Host header (the old nginx_proxy set Host: $host). It does set X-Forwarded-Proto / X-Forwarded-For.

Follow-up — janus/WebRTC media

janus / WebRTC media over the tailnet remains an unsolved follow-up (not addressed by the HTTP-serve overlay).

Babylon task #884 → deploy (facekom_dev) — dev-workflow ansible

Posted task #884 to deploy: create ansible playbooks for the fk-dev dev workflow —

  • git checkout <branch> on host /workspace/<app>,
  • docker exec <container> yarn && yarn build (pnpm for facekom_library),
  • supervisorctl restart all,
  • parameterized by branch + component subset, referencing bin/vuer.sh checkout.

Components: vuer_oss, vuer_css, esign_oss, esign_css, portal_css, facekom_library (vuer_cv excluded).

Open / next

  • Tailscale branch push to origin pending Andras’s go.
  • Three admin prereqs above must land before deploy wires onto fk-dev.
  • janus/WebRTC media path still unsolved.

Goal

Stand up a GCP VM that mirrors the on-prem FaceKom dev box (the same vuer_docker stack), reachable only over the tailnet — no DuckDNS, no public ingress. Developers hit it at <service>.facekomdev.net resolving to the box’s 100.x Tailscale IP.

Why this is mostly a no-op for the repo

Three architectural facts (verified in vuer_docker @ devel) make the migration almost entirely operational rather than code-level:

1. Host networking — nothing is published

All dev.yml services run network_mode: "host". Consequence:

  • nginx_proxy binds 0.0.0.0:80 / 0.0.0.0:443 directly on the host.
  • App processes bind 127.0.0.1:<port> (loopback only).

So once the host joins the tailnet, it’s reachable at <tailscale-ip>:443 with nothing else exposed. No per-container network wiring needed. (See the host-networking warning in Network Architecture.)

2. nginx_proxy routes by subdomain PREFIX, domain-agnostic

nginx_proxy/proxy_servers.conf routes purely on the subdomain prefix, with a regex server_name that wildcards the domain suffix:

server_name ~^oss-(.+)\.facekomdev\.net$;

Prefix → upstream port map:

PrefixUpstream (localhost)
oss-20080
css-30080
css-sdk-demo-30081
esign-oss-20180
esign-api-20181
esign-css-30180
portal-30380
cv-40080
library-50080

Because the domain suffix is matched by a wildcard regex, routing is already domain-agnostic — changing the dev domain requires no proxy_servers.conf edits.

3. One self-signed wildcard-compatible cert

TLS is a single self-signed cert /workspace/cert/dev.{crt,key} (from the external techteamer/cert repo), used by every server block. Service names are single-label (css-gcp, oss-gcp, …), so a *.facekomdev.net wildcard cert covers all of them. (See TLS Certificates.)

4. Apps build URLs from DEV_DOMAIN

Each app receives DEV_DOMAIN=<domain>.facekomdev.net (host /etc/environment → passed through each *.yml) and builds absolute URLs from it. Changing the domain is one env var, not a code change.

Only literal "duckdns" in the repo

A repo-wide search found the string duckdns in exactly one place: README.md line 17. The migration’s repo diff is therefore tiny and doc-heavy.

Phase 1 — zero repo edits

  1. Install Tailscale on the VM, join the tailnet: tailscale up --hostname=facekom-dev.
  2. Set DEV_DOMAIN=gcp.facekomdev.net (host /etc/environment).
  3. Clients add an /etc/hosts entry → the box’s 100.x tailnet IP.
  4. Reuse the existing self-signed CA (already trusted by the dev workflow).
  5. Delete DuckDNS.
  6. GCP firewall denies ALL public ingress — Tailscale works via DERP relays behind NAT, so no inbound ports are needed.

This phase touches no nginx, compose, or app files.

Phase 2 — polish (optional, still no nginx/app changes)

Superseded 2026-06-30

Phase 2a (public DNS) was retired — Decision 1 adopts Tailscale MagicDNS instead of public facekomdev.net records. The “Fallback” and “Rejected alternative” sections below were likewise overtaken by the 8b decision; see 2026-06-30 — Decisions landed + VM provisioned.

  • 2a — public DNS → private tailnet IP (RETIRED): publish a public *.facekomdev.net (or *.gcp.facekomdev.net) A-record pointing at the box’s 100.x tailnet IP. This removes the client /etc/hosts step. A CGNAT (100.64.0.0/10) address in public DNS is valid — only tailnet members can actually route to it.
  • 2b — real cert: issue a real *.facekomdev.net Let’s Encrypt wildcard via DNS-01 and drop it at /workspace/cert/dev.{crt,key}. Removes the browser warning. No nginx changes (same path, same wildcard coverage).

Fallback — no public DNS available (now the chosen path, via 8b)

Use MagicDNS + tailscale cert. The original concern was that a single MagicDNS name per node forces port/path-based routing. Decision 2 sidesteps this: instead of one node name, 8b gives each service its own MagicDNS name via a per-service tailscaled sidecar, so subdomain-prefix-style separation is preserved without proxy_servers.conf / app-URL rework (subject to the naming caveat above).

Rejected alternative (reversed 2026-06-30)

This note originally rejected a containerized Tailscale sidecar in favor of a host-level install. Decision 2 reverses that: the chosen 8b shape is precisely a per-service userspace tailscaled sidecar, because giving every service its own MagicDNS name + cert is what avoids invasive routing changes — a benefit the single host-level install can’t provide.

  • README.md — rewrite the DuckDNS / IT-DNS section.
  • install/install-tailscale.sh — new.
  • Optional cert-fetch script.
  • No nginx / compose / app changes.

Open decisions (original plan — mostly resolved 2026-06-30)

Resolution status

Tracked against the 2026-06-30 decisions. Remaining live items are consolidated under Open items (owner: Andras/user).

  1. Dev domain label — ✅ resolved: MagicDNS DEV_DOMAIN=fk-dev.taild4189d.ts.net (no public domain label).
  2. Publish public DNS → private tailnet IP? — ✅ resolved: NO (Phase 2a retired; MagicDNS instead).
  3. Self-signed vs real Let’s Encrypt cert — ✅ resolved: tailscale cert per service (requires the admin-console “HTTPS Certificates” toggle — see open items).
  4. CV / GPU in scope? — ⏳ OPEN: current VM is non-GPU (see open items).
  5. Node auth — ✅ resolved: pre-auth key reused from pmv2, ACL tag:cloud; access via ssh ops@fk-dev.
  6. VM sizing — ✅ resolved: e2-standard-4 (4 vCPU / 16 GB), 100 GB pd-balanced.
  • infrastructure — host-networking, nginx_proxy subdomain routing, TLS cert, the existing Tailscale DNS resolution chain
  • vuer_docker — compose-file overview
  • FKITDEV-8252 — concurrent vuer_docker dev-env work (UBI10 migration)
  • dev-box-cv-photo-processing-failures — hairpin-NAT gotcha on the host-networked dev box (relevant when validating inter-service HTTPS on the GCP mirror)

Graph View

Backlinks