Levandor

Babylon Deploy on polymarket-infra

11 min read

Operational notes from deploying babylon — the Rust MCP coordination server — onto the existing polymarket-infra GCP e2-small VM on 2026-06-09. Covers the host layout, the Ansible role shape, Litestream → GCS backup using GCE-native ADC, the Tailscale serve idempotency pattern, and the babylon-server CLI surface needed for token distribution. Two reusable gotchas surfaced during this deploy are captured separately in gcp-terraform-ansible-gotchas (#26 cold Rust compile time on e2-small, #27 libssl-dev vs aws-lc-rs).

For Agents

This is the operational counterpart to babylon (architecture / design) and babylon-design-decisions (the “why”). When operating or redeploying babylon on this VM, read this note. When changing the protocol or storage model, read the babylon project notes. The deploy artifacts (Ansible role babylon, workflow deploy-babylon.yml) live in the terraform repo, not the babylon repo.

What was deployed

  • Repo: wowjeeez/babylon, pinned to commit 4fe5cdb (the post-audit hardening commit referenced in babylon).
  • Target VM: polymarket-infra in europe-west3-a (e2-small, 2 vCPU shared, 2 GB RAM). The same VM already runs Docker + the polymarket-fetch containers + the OTel collector + tailscaled — see levandor-infra for the full picture.
  • Binary on disk: /opt/babylon/repo/target/release/babylon-server (built in-place on the VM from a clone of the repo).
  • SQLite database: /var/lib/babylon/babylon.db (separate from /opt/babylon/repo/ so a re-clone or rebuild can’t wipe state).
  • Systemd unit: babylon-server.service runs babylon-server serve as a long-running service.
  • Tailnet exposure: tailscale serve --bg http://127.0.0.1:8787 publishes the loopback port to the tailnet as HTTPS — tailnet-only, NOT Funnel (Funnel exposes to the public internet; serve does not).
  • Backup: Litestream continuously replicates the SQLite WAL to GCS bucket polymarket-infra-babylon-backup under path babylon.

Ansible role shape (ansible/roles/babylon)

New role added to the terraform repo alongside the existing five (base, docker, github_keys, monitoring, apps). Responsibilities, in order:

  1. OS prereqs — apt-install cmake + build-essential (needed by aws-lc-sys) + pkg-config + sqlite3 + git. Does NOT install libssl-dev — the project uses rustls with the aws-lc-rs feature, so the C build is aws-lc-sys, not openssl-sys. See [[gcp-terraform-ansible-gotchas|gotcha #27]].
  2. rustup install — pinned toolchain, idempotent via creates: on ~/.cargo/bin/rustup.
  3. Clone the repo at the pinned commit into /opt/babylon/repo.
  4. cargo build --release with creates: /opt/babylon/repo/target/release/babylon-server so re-runs short-circuit. First build on this VM size takes ~35 minutes wall-clock — see [[gcp-terraform-ansible-gotchas|gotcha #26]].
  5. Provision data dirs — /var/lib/babylon (SQLite) and /etc/babylon (config), both 0700, owned by the babylon service user.
  6. Render and start babylon-server.service (systemd unit template).
  7. Render and start litestream.service with /etc/litestream.yml pointing at the GCS bucket — no credentials_path: because Litestream picks up GCE-native ADC via the metadata server transparently (see Litestream section below).
  8. Publish over the tailnet — idempotent tailscale serve task (see Tailscale section).

Deploy workflow (deploy-babylon.yml)

Separate from deploy-apps.yml (which is for containerized app rollouts). Pattern mirrored from the existing deploy-apps.yml:

  • workflow_dispatch only (manual trigger).
  • Runner joins the tailnet via the same tag:ci OAuth client used by deploy-apps.yml.
  • Runs ansible-playbook against only the babylon role.
  • timeout-minutes: 90 — NOT the 30 minutes copy-pasted from crypto_shortterm. The first deploy attempt used 30 and got cancelled mid-cargo build. See [[gcp-terraform-ansible-gotchas|gotcha #26]] for why.

Subsequent runs are fast: the cargo task short-circuits via creates: on the binary path, so only a code change (new pin) triggers a rebuild.

Litestream → GCS using GCE-native ADC

Pattern that works under iam.disableServiceAccountKeyCreation

The GCP organization enforces constraints/iam.disableServiceAccountKeyCreation, so you cannot create a service-account JSON key for Litestream to consume. The standard Litestream + GCS recipe (write a JSON key, point credentials_path: at it) is blocked.

The workaround

Don’t try to create a key. Instead:

  1. Identify the VM’s own service account (the SA Terraform attaches to the GCE instance — see modules/vm/).
  2. Grant it roles/storage.objectAdmin scoped to the bucket (polymarket-infra-babylon-backup), not project-wide. The Terraform binding is a google_storage_bucket_iam_member resource, not a google_project_iam_member.
  3. In /etc/litestream.yml, omit credentials_path: entirely. Litestream’s GCS replica uses ADC, which on a GCE instance reads a short-lived OAuth token from the metadata server (http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token). No key on disk, no rotation burden, no env vars.

Verified working

First WAL segment + snapshot landed in the bucket within ~2s of systemctl start litestream.service. Litestream logs:

generation=564f2238597c6cb7
bucket=polymarket-infra-babylon-backup
path=babylon

Bucket configuration

  • Versioning: enabled (Litestream relies on object versions for point-in-time restore).
  • Lifecycle: prune archived versions older than 30 days AND with at least 3 newer versions, so we always retain at least 3 recent generations regardless of age.
  • Location: same region as the VM (europe-west3) to keep egress free.

When the same pattern applies

Any GCE-resident tool that needs GCS access (Litestream, gsutil, raw HTTP, the official google-cloud-storage Python lib, etc.) can use this pattern. The org policy only blocks JSON key creation; GCE-native metadata-server ADC is always available and is what google.auth.default() falls back to. Use it.

Tailscale serve — idempotent Ansible task

tailscale serve --bg http://127.0.0.1:8787 is the correct invocation to publish a loopback port to the tailnet as HTTPS:

  • serve, NOT funnel — funnel exposes the service to the public internet. serve is tailnet-only. For babylon (agents-only, not public), always serve.
  • --bg — daemonizes; the command returns immediately. Without it the call blocks the Ansible task.
  • http://127.0.0.1:8787 — loopback, the address babylon-server binds to. Tailscale handles the TLS termination on the tailnet side.

Idempotency

Calling tailscale serve repeatedly with the same arguments is idempotent in practice, but it still produces output and a non-zero “changed” signal that pollutes Ansible runs. To get a clean ok on re-runs:

  1. Query tailscale serve status --json.
  2. Parse .Web[*].Handlers["/"].Proxy and check whether http://127.0.0.1:8787 is already registered.
  3. Only re-issue tailscale serve --bg http://127.0.0.1:8787 if no handler currently proxies that target.

The babylon role’s tasks/main.yml has the working Jinja pipeline for this check.

babylon-server CLI surface (for future agent token distribution)

The babylon-server binary is multi-mode. Subcommands you will actually use during operation:

SubcommandPurpose
babylon-server serveRun the long-lived MCP server (used by the systemd unit).
babylon-server mint-token <handle>Create a token for an agent. Add --operator for elevated privileges. Prints the token plaintext to stdout — shown ONCE.
babylon-server rotate-token <handle>Issue a new token for an existing handle, invalidating the old one. The only recovery path if a token is lost.
babylon-server revoke-token <handle>Permanently disable a handle.

Tokens are shown plaintext exactly once

The mint-token output literally says “shown once; store in a 0600 EnvironmentFile as BABYLON_TOKEN”. There is no read-back endpoint — if you lose the token, your only option is rotate-token, which invalidates the old one. Distribute immediately into the consuming agent’s secret store.

Tokens minted on 2026-06-09

Seven handles were minted for the agent fleet. Each token was distributed via a per-repo GitHub Actions secret (one secret per consuming repo):

  • code
  • weather
  • hormuz
  • cryptoshort
  • bscvuln
  • deploy
  • operator (the --operator privileged token)

Token values are NOT in this vault

The plaintext token values were intentionally not written to Obsidian — they live only in the per-repo GitHub Actions secrets. If you need to know “does agent X have a token”, check the per-repo secret presence; if you need to know “what is agent X’s token”, you cannot — rotate-token and redistribute.

Sizing — babylon’s footprint on e2-small

Post-startup memory on the live VM:

ProcessPeak RSS
babylon-server~19 MB
litestream~14 MB

Combined with everything else this VM is now hosting — Docker engine + the four containerized polymarket-fetch services + the OTel collector + tailscaled + this babylon stack — there is still headroom on e2-small (2 vCPU shared, 2 GB RAM). No upsize is needed to accommodate babylon.

Resize threshold

When considering whether the VM needs an upsize, the OTel collector (:8888/metrics exposes its own resource use) and the Docker containers (which dominate this VM’s total RSS) are the variables to watch, not babylon. See spec-1-operations-runbook for the resize procedure.

Tokens + per-agent setup (2026-06-09)

Token distribution and per-agent Claude Code wiring (.mcp.json, skill install, smoke check) are documented in the babylon project folder rather than here — they’re a client/agent concern, not a VM operations concern:

  • babylon-migration — fleet cutover plan from AGENT_HANDOFF.md to babylon (phases, channel scheme, history-import decision, cutover signal)
  • babylon-operator-crib-sheet — per-agent setup: per-repo .mcp.json, secrets/babylon.env, skill install, macOS MagicDNS workaround, smoke check, register/catch_up/open_questions/open_tasks first-session calls

This note remains the source of truth for VM-side token operations (mint/rotate/revoke via babylon-server, the “shown plaintext once” rule, where the seven Phase-1 handles live).

Compose refactor (2026-06-09)

Later the same day, the bare-systemd stack (babylon + litestream as host services, tailscale serve on the host) was refactored into a 3-service docker compose project so babylon gets its own tailnet identity (separate from polymarket-infra) and the runtime libc is decoupled from the host.

Final shape:

  • 3-service compose: babylon + litestream + babylon-tailscale (sidecar).
  • Tailnet URL: https://babylon.taild4189d.ts.net/mcp — own tailnet node, distinct from the polymarket-infra node.
  • Babylon binary: still built in-place on the VM, bind-mounted into the container from /opt/babylon/repo/target/release/babylon-server (no in-image build, no registry push).
  • Runtime image: ubuntu:24.04 — matches host glibc (2.39); the binary won’t run on debian:bookworm-slim (glibc 2.36).
  • Sidecar identity: TS_HOSTNAME=babylon, no compose-level hostname:. Babylon’s perimeter check is bypassed inside the babylon container with BABYLON_ALLOW_FUNNEL=1 because the sidecar (not babylon) owns the tailnet edge.

4 failed deploy iterations before the stack came up clean. Order of discovery:

  1. RTK output leaked into gh secret set stdin — auth key stored as decorated string, tailscaled auth failed. See [[gcp-terraform-ansible-gotchas|gotcha #28]].
  2. GLIBC 2.39 vs 2.36 — bind-mounted babylon binary crashed on startup in debian:bookworm-slim. Swapped to ubuntu:24.04. See [[gcp-terraform-ansible-gotchas|gotcha #29]].
  3. Funnel-CLI safety check — babylon refused to start in-container because tailscale CLI was absent. Set BABYLON_ALLOW_FUNNEL=1. See [[gcp-terraform-ansible-gotchas|gotcha #30]].
  4. Tag rejection on --advertise-tags — sidecar’s preauth key was minted with tag:cloud, but explicit --advertise-tags=tag:cloud validates against the minting user’s tagOwners, which didn’t include them. Dropped the flag; tag inherits from the key. See [[gcp-terraform-ansible-gotchas|gotcha #32]].
  5. hostname: shadowed compose DNS — sidecar’s hostname: babylon overrode bridge-network DNS for the sibling babylon service inside the sidecar container, sending traffic to the sidecar’s own IP → 502. Dropped hostname:, kept TS_HOSTNAME only. See [[gcp-terraform-ansible-gotchas|gotcha #31]].
  6. Ansible shlex parse — -e "babylon_ts_authkey=$VAR" failed because the auth key contained shell-special bytes. Switched to -e @/tmp/vars.json via jq. See [[gcp-terraform-ansible-gotchas|gotcha #33]].

Redeploy 2026-06-25 — pin 2f6a6eb → d0f5151

Routine pin bump on the existing compose stack (Pattern A — host-managed babylon-compose.service). Dispatched via gh workflow run deploy-babylon.yml; run 28189267339 went green across all 10 steps, including the post-deploy smoke that verifies /healthz returns non-000 + all three containers (babylon, babylon-litestream, babylon-tailscale) are Up + the babylon-compose.service unit is active.

What d0f5151 brings live

  1. Issue tracker — new MCP tools surfaced: file_issue, update_issue, list_issues, get_issue, list_templates, save_template. Issue refs are #prefix-N where each channel owns a short prefix + per-channel counter. Subissues via parent: "#ref". Status flow is open → in_progress → blocked → closed.
  2. #babylon-news channel auto-subscribe — register now ensures the channel exists and subscribes each calling agent from cursor 0. Patch notes from babylon maintainers post here.
  3. Migration 0002_issues.sql — additive, idempotent via sqlx; creates issues + templates tables and adds an issue_prefix column to channels. No data loss.

Operational delta

  • Toolchain floor: rustc 1.88 (already met on the VM — no rustup bump needed).
  • No new env vars. BABYLON_ALLOWED_HOSTS unchanged. Existing /etc/babylon/babylon.env still works as-is.
  • Litestream: continued snapshotting through the bump — backups were on 2f6a6eb, now on d0f5151. Rollback to the previous pin is cheap (rebuild + restart) if needed.

Verification (post-deploy)

Called mcp__babylon__list_issues(assignee: "deploy") against the redeployed server — returned {"issues":[]}. No error → tool exists → new binary is the one serving. Sufficient as a server-side smoke; tracker semantics get exercised once agents start filing.

Plugin-side update is independent

The autopilot Phase 1 plugin update (widened auto-act protocol + babylon_guard.sh PreToolUse hook) is a separate plugin-side install: each agent runs /plugin update babylon + restarts Claude Code. The server redeploy documented above is independent of that — the new tracker tools are available regardless of plugin state.

Graph View

Backlinks