Levandor

Levandor Infra Activity Log

13 min read

For Agents

Reverse-chronological session log. Newest entries at top, grouped by date (## YYYY-MM-DD). Each bullet: one piece of work, short summary, wikilinks to docs touched. Updated by obsidian-documenter on every project doc write. Read by historian at bootstrap.

2026-07-09

  • Local OTel telemetry store on pmv2-zurich — runbook captured. SigNoz Cloud EU2 workspace expired (workspace does not exist / Invalid or missing key in otelcol-contrib journal). Rather than renew or migrate vendors, otelcol-contrib cut over to three file exporters writing JSONL to /var/log/otel/ (logs.jsonl / traces.jsonl / metrics.jsonl; 100 MB rotate, retention 7d/3d/1d, capped backups 200/100/30). Files are 0644 via systemd drop-in Umask=0022 at /etc/systemd/system/otelcol-contrib.service.d/umask.conf; ops is in otelcol-contrib group so no sudo needed for queries. DuckDB v1.1.3 installed at /usr/local/bin/duckdb; helper wrapper /usr/local/bin/otelq bakes in the OTLP-JSON FLAT_LOGS unnest CTE for common one-liners (otelq errors|recent|svc|grep|stats|services|sql|raw|disk). Verified working today: otelq services 1 → us-weather-algo=78, pmv2-cohort-algo-firstmover=27, pmv2-cohort-algo-scheduled-run=12. Pre-existing coverage gap surfaces louder now: pmv2-position-manager, pmv2-shortterm-crypto-emit/observe, pmv2-telegram-connector (Rust) do NOT export logs — tracing-opentelemetry crate has no log exporter by design; fix requires adding opentelemetry-appender-tracing bridge per service (not deploy-owned). For those services, docker logs <container> remains the log path. Traces + metrics DO flow. Ran 7-step break-glass procedure: (1) missing otelq/duckdb → re-apply monitoring role via ansible-playbook ansible/site.yml --limit pmv2-zurich --tags monitoring (NOT make deploy — that’s apps role); (2) /var/log/otel/ empty → systemctl status otelcol-contrib + journal check; (3) disk fill → rm /var/log/otel/*.jsonl.* keeps active files, permanent change in roles/monitoring/defaults/main.yml; (4) Permission denied → usermod -aG otelcol-contrib ops + re-login; (5) Binder Error: Could not find key… → OTel schema drift, patch the FLAT_LOGS CTE in roles/monitoring/tasks/main.yml and re-apply (never /usr/local/bin/otelq on the VM directly); (6) restore SigNoz/OTLP vendor → add otlp_grpc exporter alongside file exporters in roles/monitoring/templates/config.yaml.j2, stash key in secrets/signoz_ingestion_key, re-apply; (7) config source of truth = ansible/roles/monitoring/ in terraform repo, never hand-edit /etc/otelcol-contrib/config.yaml. Region context in the note: pmv2-zurich is europe-west6-a (Zurich); Supabase mgmt DB mkofmdtdldxgmmolxxhc is eu-west-1 (Dublin) — ~1400 km, 30–50 ms RTT per SELECT; PM’s gates_ms medians ~730 ms for the 4 sequential pre-book config SELECTs may reflect pool non-amortization. Also updated levandor-infra project index (agent-guides table + Related list) and terraform repo’s CLAUDE.md (new “Emergency / observability” section) with pointers to the new note — pmv2-zurich-local-otel-store, levandor-infra, observability-flow, spec-1-operations-runbook, gcp-terraform-ansible-gotchas

2026-06-25

  • weather_as DRY order-publish arm (~20:15Z): rolled apps-pmv2-weather-as-algo (Pattern B Python --loop) pin 9a07960 → e5521c4 (image sha256:8e592f03..., container restartCount=0) AND set WEATHER_AS_PUBLISH=1 in secrets/pmv2-weather-as-algo.env. New pin is env-gated, default OFF: on gate_admit, scan() publishes the pmv2-contracts entry envelope to pmv2.order.weather_as.entry once-per-market/day (self-measure dedup). Boot log: e5521c4 feat(weather-as): env-gated order publish on gate-admit (WEATHER_AS_PUBLISH, default off). Env push via rtk proxy cat secrets/pmv2-weather-as-algo.env | rtk proxy gh secret set PMV2_WEATHER_AS_ALGO_ENV -R wowjeeez/terraform (gotcha #28). Verified inside container via docker exec printenv | grep WEATHER_AS_PUBLISH. Money safety: weather_as=dry_run, global=dry_run, PM gates at build+sign — NO POST. NATS NKey already had pmv2.order.weather_as.> PUB permit from prior session (pin 5db34c0); no NATS-side change needed. First scan post-arm: 11 cities, 1 locked (paris, gate_blocked → no admit), published=0 (normal idle), settled=1. Kill switch: unset env + re-push secret + redeploy — weather-as-dry-publish-arm-2026-06-25, weather-as-deploy-pattern-b-2026-06-23
  • Cohort firstmover roll to d851d44 (~20:19Z): rolled apps-pmv2-cohort-algo-firstmover (1 of 3 cohort components — query-responder + scheduled-run untouched per cohort’s confirm) pin 8977b5b → d851d44 (image sha256:716ee0d2..., container restartCount=0). Headline change: on-chain feed migrated Alchemy WSS log-subscription → eth_getLogs HTTP polling via the 150-proxy rotating pool. Boot log: INFO cohort_algo::pmv2::watch::poll_feed: polygon poll feed armed interval_secs=5 max_blocks=2000. Kills Alchemy WSS CU bleed on the cohort lane. Same pin also bumps consensus-alert threshold 2 → 4 and resizes first_mover_core armed cohort=18 (was 20). Important architecture note: PM self-watch (pmv2-onchain-watch) is on a SEPARATE WSS path and still uses Alchemy WSS (swapped from DRPC ogws on 6/25 to escape flapping) — this redeploy only kills cohort’s WSS dep, not PM’s. Standard cohort-producer + Andras #480 standing-auth carve-out applies; no producer-DB mutation — cohort-firstmover-roll-d851d44-2026-06-25, cohort-firstmover-nats-migration
  • Crypto emit EMIT_MAX_FIRES cap clear (~19:25Z): apps-shortterm-crypto-algo-emit (Pattern B, image pmv2-crypto-algo:latest @ sha256:9f88f34f, started 2026-06-24T21:17:54Z) had been spinning ~22h on INFO emit: EMIT_MAX_FIRES reached, signal emission paused max_fires=50 at ~1Hz with ZERO JetStream publish attempts. Two-step diagnosis: (1) Crypto’s first hypothesis (DM #701) was JetStream subject-capture failure — PMV2_ORDERS maybe not capturing pmv2.order.crypto_shortterm_latency_test.>; falsified by docker exec apps-nats-nats-1 wget -qO- 'http://127.0.0.1:8222/jsz?streams=true&config=true&consumers=false' showing subjects:["pmv2.order.>"] (wildcard DOES capture). (2) Real cause: 50 pre-cutover rows in crypto_shortterm.emitted_signals (Pattern A residue, never truncated on cutover) + default EMIT_MAX_FIRES=50 (apps.yml entry + secrets/crypto-shortterm.env both omit override) hydrated the in-process total_fires counter to 50 at boot → paused state → short-circuit before publish. Critical insight: TRUNCATE alone did NOT clear the cap (watched 40s of continued spam) — the BLOCKER-3 hydration is one-shot at boot; sudo systemctl restart apps-shortterm-crypto-algo-emit.service was required to re-hydrate (INFO emit: guardrail hydrated from DB hydrated=0 confirmed). Captured cohort_publisher NKey permits (publish allow pmv2.order.crypto_shortterm_latency_test.> + $JS.API.> + $JS.ACK.>, subscribe _INBOX.>) ruling out PubAck reply-path issues. Approval chain: cryptoshort authorized via DM #705 (“GO with option A — truncate”); producer-DB mutation falls outside Andras’s #480 standing-auth carve-out (deploys/env/NATS/redispatch/restart) → surfaced for operator approval before TRUNCATE. New gotcha #38 added (in-process DB-hydrated counters only re-read on boot — TRUNCATE alone won’t reset, must bounce container). Gotcha count 37→38. Cross-linked to crypto-shortterm-emit-guardrails BLOCKER-3 hydration architecture — crypto-emit-max-fires-cap-clear-2026-06-25, gcp-terraform-ansible-gotchas, levandor-infra
  • Babylon-server redeploy (Pattern A babylon-compose.service): pin 2f6a6eb → d0f5151 via deploy-babylon.yml (run 28189267339, all 10 steps green; smoke verified /healthz non-000 + babylon/litestream/tailscale containers up + compose unit active). New d0f5151 brings live the issue tracker MCP tools (file_issue/update_issue/list_issues/get_issue/list_templates/save_template; #prefix-N refs per-channel; subissues via parent:"#ref"; status open→in_progress→blocked→closed), #babylon-news auto-subscribe on register (cursor 0 from join), and additive migration 0002_issues.sql (issues + templates tables + channels.issue_prefix column, sqlx-idempotent). Toolchain floor rustc 1.88 already met; no new env vars; BABYLON_ALLOWED_HOSTS unchanged. Verified server-side by calling list_issues(assignee:"deploy") against the redeployed server → {"issues":[]} (no error → new tool surface live). Litestream kept snapshotting through the bump so rollback is cheap. Note: the autopilot Phase 1 plugin update (widened auto-act protocol + babylon_guard.sh PreToolUse hook) is a separate plugin-side install per agent via /plugin update babylon + Claude Code restart — independent of this server redeploy — babylon-deploy-notes

2026-06-18

  • SigNoz dashboard housekeeping + NATS metrics path stood up. Infra side (uncommitted in /Users/levander/levandor/terraform, needs make configure): added prometheus-nats-exporter:0.17.3 sidecar to ansible/roles/nats/templates/compose.yml.j2 bound to 127.0.0.1:7777 with all topology flags (-varz -connz -subz -routez -gatewayz -leafz -jsz=all); added nats_exporter_image + nats_exporter_listen vars + image-pull task; added a nats Prometheus scrape job to ansible/roles/monitoring/templates/config.yaml.j2; opted four host_metrics scrapers into emitting their *.utilization series (system.cpu/filesystem/memory/paging.utilization — previously OFF by default, host dashboard CPU/FS panels had been silently empty since Spec 1); deploy-nats.yml smoke check now also asserts nats-exporter container is up and curl :7777/metrics returns. SigNoz side (live, applied via API): new dashboard polymarket-infra — NATS + JetStream UUID 019eda9d-139f-7526-b22d-c4d8a4a3cdf2 with 9 panels (scrape-up, connections, subscriptions, slow consumers, msg/byte-rates in+out, JetStream WEATHER_SIGNALS msgs+bytes, pmv2 backlog pending+ack_pending, pmv2 redeliveries); updated polymarket-infra — App services (APM) (019e6a21-...) with log-severity split graph + ERROR log-rate value panel; updated polymarket-infra — Host & Tailscale (019e6a1c-...) with disk I/O bytes + IOPS, swap paging ops, network errors+drops, tailscaled uptime, and scoped existing goroutines/DNS/DERP panels to host.name=polymarket-infra. Four new gotchas (#34 SigNoz create_dashboard HTML-escapes & → use + or update afterward; #35 SigNoz stores host metrics dotted but legacy widgets reference underscore form, both query-equivalent — prefer dotted canonical for new widgets; #36 OTel host_metrics *.utilization metrics are OFF by default and must be opted in per scraper; #37 NATS :8222 is JSON not Prometheus, needs prometheus-nats-exporter translation sidecar — do NOT expose publicly because -connz leaks client IPs and -subz leaks subject names). Gotcha count 33→37 — signoz-dashboard-housekeeping-2026-06-18, gcp-terraform-ansible-gotchas, observability-flow, levandor-infra

2026-06-09

  • Babylon migration coordination (Phase 0 done, Phase 1 in progress): captured the 4-phase cutover plan from polymarket_fetch/AGENT_HANDOFF.md (~4400 lines / ~80K tokens) to babylon (MCP server, typed messages, per-agent cursors). Phase 0 = coord round + @babylon’s Q1-Q6 answers; Phase 1 = pilot with deploy+operator; Phase 2 = code+hormuz; Phase 3 = weather+cryptoshort+bscvuln; Phase 4 = git mv AGENT_HANDOFF.md AGENT_HANDOFF.archived.md, watcher off, babylon sole channel. Logged the channel scheme (handoff/main/deploy/pm/weather/crypto/hormuz/bsc; dm: prefix reserved), history-import decision (fresh slate; @babylon files open carry-over items as tasks at cutover — weather telemetry redeploy ce4109d, crypto rustls pin, etc.), and cutover signal (authoritative decision to #main + final ## CUTOVER annotation in handoff). Updated babylon-deploy-notes with a “Tokens + per-agent setup” cross-link section pointing into the babylon project folder — babylon-migration, babylon-operator-crib-sheet, babylon-deploy-notes
  • Babylon compose refactor: bare-systemd stack (host babylon + host litestream + host tailscale serve) repackaged into a 3-service docker-compose project — babylon + litestream + babylon-tailscale sidecar. Babylon now has its own tailnet identity at https://babylon.taild4189d.ts.net/mcp (separate from polymarket-infra). Binary still built on the VM and bind-mounted from /opt/babylon/repo/target/release/babylon-server; runtime image is ubuntu:24.04 to match host glibc. Took 4 failed deploy iterations to surface every issue; order of discovery: RTK output leaking into gh secret set stdin → GLIBC 2.39 vs 2.36 mismatch on debian:bookworm-slim → babylon’s funnel-CLI safety check failing without tailscale in PATH → Tailscale rejecting explicit --advertise-tags=tag:cloud (tagOwners check) → compose hostname: shadowing sidecar DNS for sibling service → ansible -e key=$VAR shlex parsing breaking on shell-special bytes. Six new gotchas (#28–#33) captured; gotcha count is now 33. Updated babylon-deploy-notes with a “Compose refactor (2026-06-09)” section and levandor-infra overview count — babylon-deploy-notes, gcp-terraform-ansible-gotchas, levandor-infra
  • Documented the babylon (Rust MCP coord server, repo wowjeeez/babylon pin 4fe5cdb) deploy onto the existing polymarket-infra VM: new babylon Ansible role + deploy-babylon.yml workflow, binary at /opt/babylon/repo/target/release/babylon-server, SQLite at /var/lib/babylon/babylon.db, tailnet HTTPS via tailscale serve --bg http://127.0.0.1:8787, continuous backup to GCS bucket polymarket-infra-babylon-backup via Litestream. Captured the Litestream + GCE-native ADC pattern (no SA JSON key under iam.disableServiceAccountKeyCreation), the tailscale serve idempotency pattern, the babylon-server CLI surface (serve / mint-token / rotate-token / revoke-token — tokens shown plaintext once), and the sizing observation (babylon ~19 MB + litestream ~14 MB, still headroom on e2-small) — babylon-deploy-notes
  • Captured two new reusable gotchas from the same deploy: #26 cold cargo build --release on e2-small (axum + OTel + sqlx + aws-lc-rs) takes ~35 min — bump workflow timeout-minutes from 30 to 90 for any new Rust role on this VM size; and #27 libssl-dev is NOT needed when the TLS backend is rustls + aws-lc-rs (the C build is aws-lc-sys, needs cmake + build-essential instead). Updated levandor-infra overview to add the babylon row, new role, new workflow, and bump the role count from 5 → 6 — gcp-terraform-ansible-gotchas, levandor-infra, babylon-deploy-notes

2026-05-30

  • Captured a deploy gotcha (#25): the local secrets/<app>.env is gitignored and ignored by deploy-apps.yml — the GitHub Actions repo secret (e.g. POLYMARKET_FETCH_ENV) is the real source of truth. Adding POLYMARKET_PROXIES locally and dispatching GH-Actions deploys looked successful but never updated /opt/apps/<app>/.env on the VM (mtime stayed at May 27 thanks to Ansible’s idempotent content-hashed copy). Operating rule: pipe the new env file into gh secret set <SECRET> -R wowjeeez/terraform < file (reads from stdin, no value echo); source the existing content from the live VM via ssh ops@<vm> sudo cat /opt/apps/<app>/.env so working credentials aren’t overwritten by a stale local copy — gcp-terraform-ansible-gotchas

2026-05-29

  • Captured a new CD-workflow gotcha (#24): a GitHub Actions concurrency group with cancel-in-progress: false still cancels the intermediate pending run when you dispatch 3+ into the same group (one in-progress + one pending max). The middle polymarket-fetch deploy (bot) showed cancelled without ever running. Operating rule: dispatch deploys sequentially (gh run watch <id> --exit-status) or keep ≤2 in flight — gcp-terraform-ansible-gotchas

2026-05-27

  • Spec 2a Phase 1 code-complete. 16 commits on spec2a-app-deploy (terraform repo now also at github.com/wowjeeez/terraform private): AR + WIF Terraform module (modules/ar_wif/), apps Ansible role (service/job kinds, present/absent states), make deploy + make redeploy-all, reusable CI workflow (.github/workflows/build-push.yml), updated operator docs. Three new gotchas captured (OnCalendar ≠ cron, Jinja2 | precedence, ansible 2.20 include_tasks parsing) — gcp-terraform-ansible-gotchas. Phase 2 (make deprovision of us-central1 VM → reprovision in europe-west3 → first-app deploy of wowjeeez/polymarket-fetch) still pending operator action — spec-2-roadmap, gcp-app-deploy-design

2026-05-25

  • Spec 2a validation pass — 3 real bugs fixed (systemd Requires=docker.service, compose service app, app-removal lifecycle via state: absent), 4 gaps tightened (WIF HCL nesting, vm-up wording, cross-repo reusable-workflow access, output-URI resolution), 2 minor improvements (rollback, kind:/state: schema validation). Vault note re-synced — gcp-app-deploy-design
  • Spec 2a brainstorm complete — locked region europe-west3, central manifest in this terraform repo, WIF for CI auth, and service + job runtime shapes for the apps role — gcp-app-deploy-design
  • Spec 2a design committed to the repo at docs/superpowers/specs/2026-05-25-app-deploy-design.md; mirrored into the vault as gcp-app-deploy-design (Spec 2a artifact)
  • Existing VM scheduled for migration us-central1-a to europe-west3-a as part of Spec 2a (destroy + reprovision in EU; same vm_name, MagicDNS unchanged) — gcp-app-deploy-design
  • First app to ship on the new platform: wowjeeez/polymarket-fetch (telegram agent as a long-running service + snapshot job on a 15-min systemd timer) — gcp-app-deploy-design
  • Updated spec-2-roadmap status from not-started to in-design (2a only) and added the success callout pointing to the new design; updated levandor-infra overview to mark Spec 2a in-design and link the new design from Key Documents + Related
  • Wrote the agent how-to for provisioning more VMs with this Spec 1 stack, calling out the single-VM design constraint and the for_each refactor needed for N concurrent VMs — agent-guide-provision-new-vm
  • Wrote the agent how-to for configuring an application deploy onto ops-vm, with the manual path that works today and the Spec 2 features that will replace it — agent-guide-configure-app-deploy
  • Added a levandor-infra entry to the vault-root Agent Landing index, plus updated the project overview Key Documents table to link the two new agent guides — levandor-infra
  • Spec 1 deployed live on GCP — ops-vm (e2-small, us-central1-a) is online, on the tailnet, and shipping host metrics to SigNoz with zero failures — spec-1-deployment-complete
  • Captured 5 new reusable gotchas from the live deployment (Ansible gather_facts ordering, Tailscale SSH ACL + tags, OTel v0.152 receiver/exporter renames, :8888/metrics as the canonical export-success signal, rtk proxy bypass) — gcp-terraform-ansible-gotchas
  • Updated the design note to reflect the deployment, with the late refinement from key-based SSH to Tailscale SSH — gcp-vm-provisioning-design
  • Updated the project overview to mark Spec 1 deployed and Spec 2 as the active workstream — levandor-infra
  • Added a day-2 operations runbook for the live ops-vm (access, health checks, common ops, logs, cost, gotchas, secret rotation) — spec-1-operations-runbook
  • Wrote the Spec 1 implementation retrospective covering workflow, what worked, mid-stream adjustments, live-run surprises, and recommendations for Spec 2 — spec-1-retrospective
  • Drafted the Spec 2 roadmap (Artifact Registry + docker compose + Cloudflare ZTNA + batch jobs + app telemetry) with open design decisions and next-task list — spec-2-roadmap

2026-05-22

Graph View