polymarket-fetch Production Deploy (GCP / ops-vm)

How the three production workloads of polymarket-fetch run on the shared ops-vm rig in europe-west3-a (Frankfurt), live since 2026-05-27 11:08 UTC. Deploy is owned by the generic [[gcp-app-deploy-design|Spec 2a apps Ansible role]] in ~/levandor/terraform/ — this note covers the polymarket-fetch-shaped parts (services, env, verification, gotchas hit) rather than the role internals.

Services

Three units, all named with the apps-polymarket-fetch-* prefix (set by the Ansible role).

Unit	Kind	Restart	Trigger	Purpose
apps-polymarket-fetch-bot	Docker	restart=always	continuous	Telegram bot (lifecycle + milestone + per-wallet notifications)
apps-polymarket-fetch-scheduled-run	systemd timer	n/a	OnCalendar=*:0/30	30-min snapshot job — polymarket-fetch run (default top-N intersection + auto-chain diff + trades-since)
apps-polymarket-fetch-wallet-monitor	Docker	restart=always	continuous	Top-edge wallet capture rig (--top-edge-z 9 --interval-secs 5 --out-dir /var/lib/wallet-monitor) — feeds top-trader-edge / lag-arb-plan research data

Replaces the staged-but-never-loaded launchd plist (scripts/launchd/com.levander.polymarket-fetch.plist) — that path is now obsolete; the 30-min cadence is owned by the systemd timer on ops-vm, not the local Mac.

Build & deploy pipeline

git push (this repo)
  ↓
.github/workflows/release.yml
  ↓ calls reusable wowjeeez/terraform/.github/workflows/build-push.yml
  ↓ auths via GCP Workload Identity Federation (WIF)
  ↓ docker buildx build -f Dockerfile.polymarket-fetch
  ↓
Artifact Registry:
  europe-west3-docker.pkg.dev/aerobic-tesla-490112-r3/apps/polymarket-fetch:latest
  europe-west3-docker.pkg.dev/aerobic-tesla-490112-r3/apps/polymarket-fetch:<sha>
  ↓
Ansible `apps` role on ops-vm pulls + restarts the three units
  (triggered by `make deploy` or `make redeploy-all` from ~/levandor/terraform/)

Dockerfile.polymarket-fetch is multi-stage: rust:1-bookworm builder → debian:bookworm-slim runtime.

Host

• VM: ops-vm, GCP europe-west3-a, e2-small, 50GB pd-balanced
• External IP: 34.179.237.88
• Tailscale hostname: ops-vm-1 (auto-suffixed — see gotcha #15 below; should restore to ops-vm once the offline US node is cleaned up in the Tailscale admin console)
• Operator entry points: make ssh / make deploy / make redeploy-all in ~/levandor/terraform/

Branch state

Phase 1+2 of Spec 2a was validated by this live deploy on branch spec2a-app-deploy in ~/levandor/terraform/ (16 commits). Not yet merged to main. Until the merge lands, treat main as out-of-date for app-deploy work.

Environment

All three units share the same env-file pattern owned by the apps role. The polymarket-fetch-specific values:

• SUPABASE_DB_URL — must be the session pooler URL: postgresql://postgres.<ref>:<pw>@aws-1-eu-west-1.pooler.supabase.com:5432/postgres. The direct DSN (db.<ref>.supabase.co) is IPv6-only and will fail from inside Docker — see gotcha #18 below.
• TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID — required for the bot unit; the scheduled-run + wallet-monitor units also fire lifecycle notifications if set.
• Wallet-monitor-specific args are passed as the container CMD: --top-edge-z 9 --interval-secs 5 --out-dir /var/lib/wallet-monitor. JSONL output lives on the host at /var/lib/wallet-monitor/.

Live verification (as of 2026-05-27 ~14:10 UTC)

Unit	Signal	Value
scheduled-run	ticks since deploy	10
scheduled-run	ticks in last 90 min	6
scheduled-run	latest pm_runs.started_at	2026-05-27 13:57:06 UTC
wallet-monitor	rows captured in last 30 min	2,668
wallet-monitor	latest pm_trades.trade_time	2026-05-27 14:07:22 UTC
wallet-monitor	on-disk at /var/lib/wallet-monitor/	1.4M across 2 JSONL files
bot	systemctl is-active	active
all three	systemctl is-active	active

Sources: Supabase MCP for DB-side counts; Tailscale SSH to ops-vm-1 for on-disk + systemd state.

Bot end-to-end not yet smoke-tested

The bot container is active per systemd, but no Telegram command has been sent against the deployed instance to confirm the round-trip. Open follow-up.

DB state at deploy time

• pm_runs — 212 historical rows (from prior manual / local runs) + 10 new rows from the 30-min timer since 11:08 UTC.
• pm_trades — growing live from wallet-monitor (2,668 in last 30 min at observation time).
• No backfill of historical pm_runs performed; optional follow-up if research wants the pre-deploy series stitched in.

Gotchas hit during this deploy

All four are documented in detail in gcp-terraform-ansible-gotchas (#15-#18). Summary of impact on polymarket-fetch:

#18 Supabase IPv6 vs Docker IPv4 (the killer)

All three containers crash-looped on first deploy with Network is unreachable (os error 101) from sqlx. Root cause: Supabase direct DSN is IPv6-only, Docker default bridge is IPv4-only. Fix: switched SUPABASE_DB_URL in the deployed env file to the session pooler form (aws-1-eu-west-1.pooler.supabase.com:5432). Transaction pooler (:6543) is not sqlx-compatible — must be session.

For Agents

This pattern applies to every future Supabase-using app deployed via Docker on this rig. If a new app fails with “Network is unreachable” against Postgres, the first thing to check is whether its env points at the direct DSN.

#17 .dockerignore killed sqlx::migrate!()

The repo’s .dockerignore excluded supabase/, but the binary embeds sqlx::migrate!("../../supabase/migrations") at compile time — cargo build inside the Docker builder couldn’t see the migrations directory and failed at the macro. Fix: commit 834bc66 (fix(docker): include supabase/migrations/ in build context) — dropped the supabase/ exclusion.

#16 gcloud APT GPG key needs .asc extension

Ansible’s gcloud-SDK repo task hung in retry loops because the key was saved as .gpg but the served content is ASCII-armored. Apt refused to parse it as a binary keyring. Fix on the infra side (commit 4e6a0f1 in the terraform repo) — not a polymarket-fetch change, but blocked this deploy until resolved.

#15 Tailscale name collision

terraform destroy of the old US ops-vm left the node alive in the tailnet. When the new EU VM came up and ran tailscale up, the canonical hostname was taken — Tailscale auto-suffixed to ops-vm-1. Ansible inventory still said ansible_host=ops-vm, MagicDNS resolved to the offline old node, SSH timed out. Fix: patched inventory at runtime to ops-vm-1. Manual cleanup pending: delete the offline ops-vm node in the Tailscale admin console at https://login.tailscale.com/admin/machines, then re-register the new VM under the freed name. Until then, make ssh needs the patched inventory.

Open follow-ups

• Telegram bot end-to-end smoke test (send a command, verify response)
• Tailscale admin cleanup of the offline ops-vm node so make ssh works without the inventory patch
• Merge spec2a-app-deploy to main in ~/levandor/terraform/
• (Optional) Backfill old pm_runs rows pre-deploy if the research track wants continuity

NOT deployed (intentionally out of scope)

• pm-arb validate — paper-trade rig for the latency-arb thesis. 6 known blocker bugs being fixed in a separate track (state HashMap leak, slug-parse ghost state, config defaults mismatch with backtest, stale-book ghost fills, Supabase silent fallback). See lag-arb-plan.
• pm-arb live / pm-arb paper — not yet built (per master spec docs/lag-arb-plan.md M1/M2 phases).
• lag-probe — separate workload (read-only book-lag capture), separate deploy decision. See lag-probe.

Related

• polymarket-fetch — main project overview
• wallet-monitor — the rig itself (data + analysis it feeds)
• scheduled-run — the 30-min snapshot job’s role in the pipeline
• top-trader-edge — strategy v2 thesis that wallet-monitor exists to feed
• lag-arb-plan — master spec for the (separate) latency-arb workstream
• ci — GitHub Actions pipeline that builds the deployed images
• Infra side (do not duplicate here, but cross-reference):
  • gcp-terraform-ansible-gotchas — gotchas 15-18 hit during this deploy
  • gcp-app-deploy-design — Spec 2a design for the apps role
  • spec-2-roadmap — broader roadmap this deploy executed against