Levandor

Levandor Infra Topics

14 min read

For Agents

Living index of themes for this project. Each H2 is a topic; bullets are wikilinks to related notes. Updated by obsidian-documenter when documenting work. Read by historian at bootstrap. Topics kept alphabetical.

Agent Guides

Artifact Registry & CI

  • gcp-app-deploy-design — Spec 2a Terraform modules/ar_wif/ (AR Docker repo apps in europe-west3 + WIF pool/provider + ci-pusher SA + per-repo IAM bindings); reusable build-push.yml GitHub Actions workflow called via workflow_call from each app repo
  • gcp-terraform-ansible-gotchas — deploy-apps.yml concurrency group with cancel-in-progress: false cancels the intermediate pending run when 3+ deploys are dispatched at once; dispatch sequentially (gotcha #24)
  • gcp-terraform-ansible-gotchas — deploy-apps.yml “Materialize per-app env file” step reads the GitHub Actions repo secret, not the local gitignored secrets/<app>.env; local edits never reach GH-Actions-driven deploys (gotcha #25)

Ansible Configuration

  • gcp-app-deploy-design — Spec 2a adds the apps role (compose + systemd templates) and extends the docker role with the AR credential helper
  • gcp-vm-provisioning-design — the four-role Ansible layer (base, docker, github_keys, monitoring), ansible.cfg loading, explicit fact-gathering after wait_for_connection
  • gcp-terraform-ansible-gotchas — ansible.cfg auto-load gotcha, gather_facts ordering gotcha
  • spec-1-deployment-complete — Ansible roles all green against live ops-vm
  • spec-1-operations-runbook — re-applying Ansible via make configure on the live VM
  • babylon-deploy-notes — new babylon role (rustup install + cargo build with creates: short-circuit + systemd + litestream + idempotent tailscale serve), brings the role count from 5 to 6

Backup & DR

  • babylon-deploy-notes — Litestream continuous SQLite WAL replication to GCS bucket polymarket-infra-babylon-backup (versioned, 30d + ≥3 newer-versions lifecycle); GCE-native ADC instead of SA JSON key (org policy iam.disableServiceAccountKeyCreation blocks key creation)

App Deploy

  • gcp-app-deploy-design — Spec 2a approved design: AR + WIF + Ansible apps role (service and job runtime shapes) + central manifest + EU migration + first app (polymarket-fetch)
  • spec-2-roadmap — Artifact Registry + docker compose + Cloudflare ZTNA + batch jobs + app logs/traces; the application-deploy layer on top of ops-vm
  • agent-guide-configure-app-deploy — manual deploy path today (/opt/apps/<name>, named compose projects, OTel-via-localhost) and what Spec 2 replaces it with
  • gcp-terraform-ansible-gotchas — GH-Actions deploys read env values from the repo secret, not the local file; “successful” deploys can silently leave the /opt/apps/<app>/.env mtime untouched for days because Ansible’s copy is content-hashed (gotcha #25)
  • crypto-emit-max-fires-cap-clear-2026-06-25 — Pattern A → B cutover residue: post-cutover containers can inherit DB-state that hydrates into in-process safety counters at boot, leaving the new service in a “paused” state until manually cleared (TRUNCATE + container restart). Worth checking on ANY Pattern A → B cutover involving DB-hydrated guardrails
  • cohort-firstmover-roll-d851d44-2026-06-25 — apps-pmv2-cohort-algo-firstmover pin 8977b5b → d851d44: on-chain feed migrated Alchemy WSS log-subscription → eth_getLogs HTTP polling via the 150-proxy rotating pool (kills cohort WSS CU bleed); consensus-alert threshold 2 → 4; PM self-watch (pmv2-onchain-watch) is on a SEPARATE WSS path and STILL uses Alchemy WSS — don’t conflate the two lanes
  • weather-as-dry-publish-arm-2026-06-25 — apps-pmv2-weather-as-algo pin 9a07960 → e5521c4 + env arm WEATHER_AS_PUBLISH=1: first time the weather-AS lane has a path to publish entry envelopes to NATS pmv2.order.weather_as.entry; env-gated default OFF, once-per-market/day dedup via self-measure ledger; money safety intact (dry_run end-to-end, PM gates at build+sign); kill switch = unset env + re-push secret + redeploy

Cost & Sizing

Docker & Compose

  • babylon-deploy-notes — 3-service compose (babylon + litestream + babylon-tailscale sidecar), bind-mounted binary, ubuntu:24.04 runtime image to match host glibc
  • gcp-terraform-ansible-gotchas — GLIBC mismatch between Ubuntu 24.04 host and debian:bookworm-slim runtime (gotcha #29); compose hostname: overrides bridge-network DNS for the matching name inside the container (gotcha #31)
  • signoz-dashboard-housekeeping-2026-06-18 — prometheus-nats-exporter sidecar added to the nats compose project (2026-06-18); connects to NATS over the compose-internal docker network at http://nats:8222, exposes Prometheus metrics on 127.0.0.1:7777 for the host-native OTel collector to scrape

EU Migration

  • gcp-app-deploy-design — Spec 2a migrates the existing ops-vm from us-central1-a to europe-west3-a (destroy + reprovision; same vm_name so MagicDNS resolution stays unchanged)

Gotchas & Learnings

  • gcp-terraform-ansible-gotchas — thirty-eight reusable GCP / Terraform / Ansible / OTel / RTK / GitHub Actions / Rust / Tailscale / Docker / SigNoz / NATS / process-lifecycle traps from Spec 1 validation, live deployment, Spec 2a/2b execution, the CD workflow rollout, the GH-secret vs local-env divergence, the 2026-06-09 babylon Rust deploy, the babylon compose refactor (sidecar pattern), the 2026-06-18 SigNoz/NATS housekeeping (#34–#37), and the 2026-06-25 crypto emit cap clear (#38 in-process DB-hydrated counters only re-read on boot)
  • crypto-emit-max-fires-cap-clear-2026-06-25 — Pattern A → B cutover residue case study: 50 pre-cutover rows in crypto_shortterm.emitted_signals + default EMIT_MAX_FIRES=50 left the new container in a paused boot state for ~22h; TRUNCATE + container restart fixed it
  • spec-1-operations-runbook — the three gotchas most likely to bite on a re-run, with cross-links to the full reference
  • spec-1-retrospective — meta-reflection on which gotchas the two-stage review caught vs which surfaced only at deployment time
  • babylon-deploy-notes — where gotchas #26 (cold Rust compile time ~35 min on e2-small → bump workflow timeout to 90 min) and #27 (libssl-dev not needed with rustls + aws-lc-rs; use cmake instead) surfaced
  • signoz-dashboard-housekeeping-2026-06-18 — where gotchas #34 (SigNoz create_dashboard HTML-escapes &), #35 (host metrics dotted-vs-underscore form), #36 (OTel host_metrics *.utilization series OFF by default), #37 (NATS :8222 is JSON, not Prometheus — needs prometheus-nats-exporter translation sidecar) surfaced

How-to / Agent Guides

Local OTel Store (post-SigNoz)

  • pmv2-zurich-local-otel-store — SigNoz Cloud EU2 workspace expired; otelcol-contrib on pmv2-zurich cut over to three file exporters writing JSONL to /var/log/otel/ (logs 7d/200 backups, traces 3d/100, metrics 1d/30, 100 MB rotate). DuckDB v1.1.3 + otelq wrapper (errors|recent|svc|grep|stats|services|sql|raw|disk) for queries. Files are 0644 via systemd Umask=0022 drop-in; ops in otelcol-contrib group — no sudo needed. Includes 7-step break-glass runbook: missing tools → re-apply monitoring role (NOT make deploy); empty files → systemctl + journal; disk fill → rm rotated backups + retention knob in roles/monitoring/defaults/main.yml; Permission denied → usermod -aG otelcol-contrib ops; Binder Error → patch FLAT_LOGS CTE in roles/monitoring/tasks/main.yml (never /usr/local/bin/otelq on VM); restore OTLP vendor → dual exporter in config.yaml.j2 + secrets/signoz_ingestion_key; config source of truth = ansible/roles/monitoring/ in terraform repo
  • pmv2-zurich-local-otel-store — pre-existing coverage gap re-surfaced: pmv2-position-manager, pmv2-shortterm-crypto-emit/observe, pmv2-telegram-connector (Rust) don’t export logs (their tracing-opentelemetry crate has no log exporter by design; fix = add opentelemetry-appender-tracing bridge in each service repo). Traces + metrics DO flow. For those services, docker logs <container> is still the log path
  • pmv2-zurich-local-otel-store — Zurich↔Dublin latency context: pmv2-zurich is GCP europe-west6-a; Supabase mgmt DB mkofmdtdldxgmmolxxhc is eu-west-1 (~1400 km, 30–50 ms RTT per SELECT); PM gates_ms medians ~730 ms across 4 sequential config SELECTs pre-book-fetch may reflect pool non-amortization

NATS & JetStream

  • signoz-dashboard-housekeeping-2026-06-18 — prometheus-nats-exporter:0.17.3 sidecar in the nats compose project translates NATS’s JSON :8222 endpoints into a Prometheus scrape target on 127.0.0.1:7777; OTel collector picks it up via its prometheus receiver; new SigNoz polymarket-infra — NATS + JetStream dashboard (UUID 019eda9d-139f-7526-b22d-c4d8a4a3cdf2) keys off these metrics — pmv2 consumer backlog (pending + ack_pending) is the headline panel for “is the consumer keeping up”
  • observability-flow — NATS Prometheus metrics path documented alongside host/Tailscale/traces/logs as the fifth signal source feeding the OTel collector
  • gcp-terraform-ansible-gotchas — NATS :8222 is JSON, NOT Prometheus — needs prometheus-nats-exporter translation; -jsz=all flag is required for JetStream stream/consumer metrics; do NOT expose the exporter port publicly (-connz leaks client IPs, -subz leaks subject names) (gotcha #37)
  • crypto-emit-max-fires-cap-clear-2026-06-25 — JSM stream subject-capture proof recipe: docker exec apps-nats-nats-1 wget -qO- 'http://127.0.0.1:8222/jsz?streams=true&config=true&consumers=false' returns the full stream config (subjects: [...]) + state (last_seq) from inside the nats container without admin NKey CLI access; useful when ruling NATS in/out as the cause of “no publishes are landing”; also captures the cohort_publisher NKey publish-permit set (allow pmv2.order.crypto_shortterm_latency_test.> + $JS.API.> + $JS.ACK.>, sub _INBOX.>)
  • weather-as-dry-publish-arm-2026-06-25 — first publisher on subject pmv2.order.weather_as.entry (env-gated by WEATHER_AS_PUBLISH, default OFF); weather_as NKey already had PUB permit on pmv2.order.weather_as.> from prior session (pin 5db34c0); subject is captured by PMV2_ORDERS wildcard pmv2.order.> — no NATS-side change needed for the arm

Networking & Tailscale

  • gcp-vm-provisioning-design — dedicated custom VPC, no public SSH, Tailscale MagicDNS access, tag:cloud, Tailscale SSH ACL
  • gcp-terraform-ansible-gotchas — default VPC open SSH rule, OS Login override, Tailscale SSH ACL autogroup:self vs tagged devices; sidecar --advertise-tags rejection (tagOwners not key, gotcha #32); compose hostname: shadowing sibling-service DNS in sidecar containers (gotcha #31); babylon’s funnel-CLI safety check needs BABYLON_ALLOW_FUNNEL=1 when sidecar owns the tailnet edge (gotcha #30)
  • spec-1-deployment-complete — late refinement from key-based SSH to Tailscale SSH; ACL targets tag:cloud
  • spec-1-operations-runbook — ssh ops@ops-vm and tailscale ssh ops@ops-vm access paths
  • babylon-deploy-notes — tailscale serve --bg http://127.0.0.1:8787 to publish a loopback port as HTTPS on the tailnet (NOT funnel; that exposes to the public internet); idempotent Ansible task pattern via tailscale serve status --json + handler check; compose refactor (2026-06-09) gives babylon its own tailnet identity babylon.taild4189d.ts.net via a Tailscale sidecar (separate from polymarket-infra)

Operations

  • spec-1-operations-runbook — day-2 access, health checks, common operations (re-apply / resize / teardown), logs, where things live, cost monitoring, secret rotation
  • agent-guide-configure-app-deploy — day-1 app deploy hygiene that survives into Spec 2 (per-app folders, named compose projects, localhost-bound ports)
  • pmv2-zurich-local-otel-store — day-2 query surface for the local OTel store on pmv2-zurich: otelq errors|recent|svc|grep|stats|services|sql|raw|disk, no sudo required, all output is JSONL under /var/log/otel/
  • babylon-deploy-notes — babylon operational layout on polymarket-infra (binary / SQLite / systemd / litestream / tailscale serve paths), sizing footprint (~19 MB + ~14 MB), babylon-server CLI surface (mint/rotate/revoke tokens), and the “tokens shown plaintext once” rule
  • babylon-migration — fleet cutover from AGENT_HANDOFF.md to babylon (4 phases over 24-48h; Phase 0 done, Phase 1 pilot in progress with deploy+operator); channel scheme; carry-over-as-tasks history-import decision
  • babylon-deploy-notes — 2026-06-25 redeploy: babylon-compose.service pin 2f6a6eb → d0f5151 via deploy-babylon.yml run 28189267339; brings issue tracker MCP tools, #babylon-news auto-subscribe, and additive migration 0002_issues.sql live; no env-var changes; verified by list_issues returning {"issues":[]}

Process

  • spec-1-retrospective — RPI-style brainstorm → spec → plan → execute → validate workflow with subagent-driven-development; what worked, what needed mid-stream adjustment, surprises during live run

Provisioning & Design

  • gcp-app-deploy-design — Spec 2a design: extends the Terraform root with modules/ar_wif/, adds the apps Ansible role, migrates the deployment to europe-west3
  • levandor-infra — project overview, two-spec roadmap, Spec 1 deployed
  • gcp-vm-provisioning-design — approved Spec 1: Terraform-provisions / Ansible-configures VM lifecycle
  • spec-1-deployment-complete — live deployment state: ops-vm e2-small in us-central1-a, on the tailnet, Docker + fail2ban + OTel
  • agent-guide-provision-new-vm — agent walkthrough for make preflight → plan → provision → verify against this Terraform root, plus the multi-VM refactor constraint
  • babylon-deploy-notes — operational layout of the babylon Rust MCP service on polymarket-infra (separate Ansible role + dedicated CD workflow alongside the containerized-apps pipeline)

Roadmap

  • gcp-app-deploy-design — Spec 2a approved (core app deploy + EU migration + first app)
  • spec-2-roadmap — application-deploy layer (Artifact Registry, docker compose, Cloudflare ZTNA, batch jobs, app telemetry); open design decisions; next-task list

Rust on the VM

  • babylon-deploy-notes — building axum + OpenTelemetry + sqlx + aws-lc-rs in-place on polymarket-infra (e2-small) via the babylon Ansible role; cargo task gated on creates: <binary path> so re-runs short-circuit
  • gcp-terraform-ansible-gotchas — gotcha #26 cold-build wall-clock (~35 min on e2-small) and the 90-min workflow timeout it forces; gotcha #27 libssl-dev vs cmake apt prereq divergence between openssl-sys and aws-lc-sys crates (use cargo tree -i to confirm before writing the apt list)

Secrets & Auth

  • gcp-app-deploy-design — Spec 2a uses Workload Identity Federation (OIDC) for CI to AR auth (no long-lived service-account keys); per-app secrets/<app>.env files, gitignored, mode 0600, Ansible-copied to /opt/apps/<app>/.env
  • gcp-vm-provisioning-design — ADC auth, Tailscale auth key with tag:cloud, SigNoz ingestion key, passwordless operation
  • spec-1-deployment-complete — Tailscale SSH replaces SSH keypair + macOS Keychain step entirely
  • spec-1-operations-runbook — secret rotation procedures for the Tailscale auth key and SigNoz ingestion key
  • gcp-terraform-ansible-gotchas — for apps deployed via deploy-apps.yml, the GitHub Actions repo secret (e.g. POLYMARKET_FETCH_ENV) is the source of truth, not the local gitignored secrets/<app>.env; rotate env vars by piping a file into gh secret set -R <repo> < file then re-dispatching the workflow (gotcha #25)
  • babylon-deploy-notes — under org policy iam.disableServiceAccountKeyCreation, Litestream → GCS auths via the VM’s own service-account ADC through the GCE metadata server (no JSON key, no credentials_path: in /etc/litestream.yml); bucket-scoped roles/storage.objectAdmin grant on the VM SA is sufficient. Same pattern applies to any GCE-resident GCS client. Babylon agent tokens are minted by babylon-server mint-token <handle> and shown plaintext once — no read-back, lost token requires rotate-token; values distributed via per-repo GH Actions secrets, not in this vault

SigNoz Dashboards

  • signoz-dashboard-housekeeping-2026-06-18 — 2026-06-18 housekeeping pass: new polymarket-infra — NATS + JetStream dashboard (UUID 019eda9d-139f-7526-b22d-c4d8a4a3cdf2, 9 panels including pmv2 backlog pending+ack_pending and redeliveries); updated polymarket-infra — App services (APM) (UUID 019e6a21-9eee-7e14-9603-1e00ebd59194) with log-severity split and ERROR log-rate panels; updated polymarket-infra — Host & Tailscale (UUID 019e6a1c-5be5-7383-83ff-23f8562038f4) with disk I/O bytes+IOPS, swap paging ops, network errors+drops, tailscaled uptime, and host.name=polymarket-infra scoping on the Tailscale-internal panels
  • observability-flow — three production dashboards (host & Tailscale, App services APM, NATS + JetStream) all live in SigNoz Cloud EU2; reach via https://eu2.signoz.cloud/dashboard/<uuid>
  • gcp-terraform-ansible-gotchas — SigNoz create_dashboard HTML-escapes & in titles, use + or update_dashboard afterward (gotcha #34); host metrics stored under dotted canonical names but legacy widgets reference underscore form, both query-equivalent (gotcha #35)

Telemetry & Monitoring

  • gcp-vm-provisioning-design — OpenTelemetry Collector host metrics to SigNoz Cloud (eu2)
  • gcp-terraform-ansible-gotchas — OTel :8888/metrics as export-success signal, v0.152 receiver/exporter renames; OTel host_metrics *.utilization series OFF by default (gotcha #36); NATS :8222 is JSON not Prometheus, needs prometheus-nats-exporter translation sidecar (gotcha #37)
  • spec-1-deployment-complete — ~25k host-metric points sent to ingest.eu2.signoz.cloud, zero failures
  • spec-1-operations-runbook — curl :8888/metrics health-check and OTel log location for live debugging
  • spec-2-roadmap — extending otelcol-contrib with filelog for logs and OTLP receivers for traces
  • agent-guide-configure-app-deploy — manual recipe for extending the collector config with otlp + filelog receivers for app logs/traces before Spec 2 lands
  • signoz-dashboard-housekeeping-2026-06-18 — NATS Prometheus metrics path stood up via prometheus-nats-exporter:0.17.3 sidecar (bound to 127.0.0.1:7777, flags -varz -connz -subz -routez -gatewayz -leafz -jsz=all); four host_metrics *.utilization scrapers explicitly opted in (CPU/FS/memory/paging — host dashboard CPU+FS panels had been silently empty since Spec 1)
  • pmv2-zurich-local-otel-store — post-SigNoz-expiry cutover on pmv2-zurich: three file exporters write JSONL to /var/log/otel/ (100 MB rotate, retention 7d/3d/1d, capped backups 200/100/30), queried via DuckDB + otelq wrapper baking the OTLP-JSON FLAT_LOGS unnest CTE; files are 0644 via systemd Umask=0022 drop-in so no sudo needed. Sixth OTel signal source description alongside host / Tailscale / NATS / traces / logs

Tooling & Workflow

  • gcp-terraform-ansible-gotchas — rtk proxy bypasses RTK output filtering when raw output is load-bearing (Terraform/Ansible/curl); gh workflow run concurrency-group dispatch race (gotcha #24); RTK output leaks into gh secret set stdin when piping — always rtk proxy gh secret set … (gotcha #28); ansible -e key=$VAR shlex-parse breaks on shell-special bytes, use -e @file.json built via jq (gotcha #33)

Troubleshooting

  • spec-1-operations-runbook — common gotchas section keyed to the three most likely re-run traps; logs commands for OTel / fail2ban / Docker
  • crypto-emit-max-fires-cap-clear-2026-06-25 — symptom: container stuck logging EMIT_MAX_FIRES reached, signal emission paused max_fires=N at ~1Hz; diagnosis path (wrong hypothesis first — JetStream subject capture, ruled out via /jsz?streams=true&config=true); real cause (DB-hydrated guardrail + Pattern A residue); fix (TRUNCATE + systemctl restart); verification (boot log guardrail hydrated from DB hydrated=0, DB row growth, JetStream last_seq advance)
  • pmv2-zurich-local-otel-store — 7-step break-glass for the local OTel store: missing otelq/duckdb → re-apply monitoring role (NOT make deploy); empty /var/log/otel/ → systemctl status otelcol-contrib + journal; disk fill → rm /var/log/otel/*.jsonl.* keeps active files (perm change in roles/monitoring/defaults/main.yml); Permission denied → usermod -aG otelcol-contrib ops + re-login; DuckDB Binder Error: Could not find key… → OTel schema drift, patch FLAT_LOGS CTE in roles/monitoring/tasks/main.yml (never on VM); restore SigNoz/OTLP vendor → dual exporter in config.yaml.j2 alongside file exporters; config source of truth = ansible/roles/monitoring/

Workflow

  • spec-1-retrospective — two-stage subagent review (spec-compliance + code-quality) caught four design-stage bugs; carryover recommendations for Spec 2

Graph View