Agent Guide — Redeploy an App

For LLM agents

Use this when an app already exists in deploys/apps.yml and you need to roll the container to a newer image (typically because CI just rebuilt). The fastest path is the GitHub Actions CD workflow; the laptop-side fallback works too.

What “redeploy” means here

The app’s manifest entry stays the same. The container is bounced; ExecStartPre=docker pull <image> fetches whatever :latest currently points to in Artifact Registry. For kind: service entries, systemd’s state: restarted always restarts. For kind: job, the next timer firing uses the new image.

Path 1 — GitHub Actions CD (preferred)

Trigger from anywhere, no laptop state needed:

gh workflow run deploy-apps.yml -R wowjeeez/terraform --ref main -f app=<app_name>

Or “Run workflow” in the Actions tab, pick the app from the dropdown.

Valid app values come from the workflow’s inputs.app.options:

• all — redeploys every entry in deploys/apps.yml (use sparingly: it bounces services regardless of need)
• polymarket-fetch-bot
• polymarket-fetch-scheduled-run
• polymarket-fetch-wallet-monitor
• pm-arb-validate

(If you add a new app, also update the workflow’s options: list — otherwise the dropdown won’t list it.)

The workflow does, in order:

1. Checkout the terraform repo at main.
2. Connect to Tailscale (OAuth client → ephemeral tag:ci auth key, killed at workflow end).
3. Install Ansible from apt.
4. Materialize secrets/polymarket-fetch.env from the POLYMARKET_FETCH_ENV GitHub Actions secret (mode 0600).
5. Resolve the VM’s tailnet IP into /etc/hosts (userspace-mode tailscaled has no MagicDNS — see gcp-terraform-ansible-gotchas #19 for the trap).
6. Write ~/.ssh/config with the SOCKS5 ProxyCommand routing through 127.0.0.1:1055.
7. Run make deploy APP=<name> (or make redeploy-all for app=all).
8. Smoke check: ssh ops@polymarket-infra 'sudo systemctl is-active apps-<name>.{service,timer}'.

Watch the run: gh run watch <RUN_ID> -R wowjeeez/terraform. ETA ~90s end-to-end.

Path 2 — Operator laptop

If Tailscale is up locally and ADC is fresh:

cd /Users/levander/levandor/terraform
make deploy APP=<app_name>          # one app
make redeploy-all                   # everything

Same Ansible role runs, just with the laptop-side inventory.

Verifying the redeploy

After the workflow completes, three checks:

gh workflow run ... && gh run watch <id>     # Path 1 only — must exit 0
 
# Verify systemd unit is active:
ssh ops@polymarket-infra 'sudo systemctl is-active apps-<name>.service apps-<name>.timer'
 
# Verify the new image digest is the one being run:
ssh ops@polymarket-infra 'sudo docker inspect apps-<name> --format "{{.Image}} {{.State.StartedAt}}"'

StartedAt should be within the last minute. Image digest should match the most recent gcloud artifacts docker images list europe-west3-docker.pkg.dev/aerobic-tesla-490112-r3/apps/<image_name> --include-tags --limit=2.

For SigNoz: a fresh redeploy of a kind: service produces a momentary gap then resume in the per-service trace + log rate. Check [[#dashboards|Dashboard #2 (APM)]] — call rate should dip toward zero for ~10s, then rebound.

Common failure modes

Symptom	Cause	Fix
Workflow fails at “Connect to Tailscale” with Status: 403, "calling actor does not have enough permissions"	OAuth client missing tag:ci in its Tags field, or ACL missing tagOwners for tag:ci	See gcp-terraform-ansible-gotchas #20 (OAuth client tag scoping)
“Could not resolve hostname polymarket-infra: name resolution”	userspace tailscaled has no MagicDNS	Workflow handles this via fallback IP — if you see this, the IP changed (e.g., VM rebuilt) and the fallback 100.73.121.76 in the workflow needs updating
Connection closed by UNKNOWN port 65535	SSH ACL missing tag:ci → tag:cloud users:ops	Add to the ssh: block in admin → Access controls
nc: connect to 127.0.0.1 port 1055: Connection refused	Tailscale SOCKS5 proxy not enabled	Workflow does this via tailscaled-args: '-socks5-server=127.0.0.1:1055' — if you see this, that input is missing or the action regressed
Ansible “ok=3 changed=0” with no _present.yml tasks	include_tasks tag propagation bug (fixed by apply: tags: [apps])	Already fixed at cccd438; if it reappears it’s a regression
make deploy fails on systemd-analyze calendar for a job	schedule: in manifest is cron syntax, not OnCalendar	See gcp-terraform-ansible-gotchas #12
Container starts then crash-loops with Network is unreachable (os error 101) reaching Supabase	IPv6-only Supabase direct connection	Switch SUPABASE_DB_URL to the session pooler (port 5432, aws-1-<region>.pooler.supabase.com) — see gcp-terraform-ansible-gotchas #18

When to redeploy vs not

• Image bumped automatically on push to main — every commit to the app repo’s tracked paths kicks off release.yml which rebuilds + pushes :latest. Containers do NOT auto-pull. You must explicitly redeploy.
• Manifest entry changed (e.g., new env var, schedule change, command flag) — must redeploy to render the new compose file.
• Just bouncing because of a hang — ssh ops@polymarket-infra 'sudo systemctl restart apps-<name>.service' is faster than the full workflow.

Related

• agent-guide-configure-app-deploy — deploying a brand-new app
• agent-guide-remove-app — clean teardown
• agent-guide-manage-services — day-2 ops
• observability-flow — what to look for in SigNoz post-deploy
• gcp-terraform-ansible-gotchas
• levandor-infra