Levandor

Agent Guide — Manage Running Services (Day-2 Ops)

5 min read

For LLM agents

Operator cheat sheet for the live VM. SSH access is via Tailscale only; the user ops has passwordless sudo. All app containers are managed by systemd units named apps-<app_name>.{service,timer} and live in /opt/apps/<app_name>/.

Connect

ssh ops@polymarket-infra

(Or ssh ops@ops-vm-1 if the rename’s Tailscale cleanup isn’t done yet — see gcp-terraform-ansible-gotchas #15.)

Use /usr/bin/ssh if your shell wraps the ssh command in some way (zsh aliases, etc.).

Quick status — all apps

sudo systemctl list-units 'apps-*' --no-pager

Output is a table of unit name, load state (loaded/not-found), active state (active/failed/inactive), sub-state (running/dead/start-pre/…).

For containers specifically:

sudo docker ps --filter 'name=apps-' --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'

Healthy steady-state (post Spec 2a+2b, ~2026-05-27):

NAMES                                  STATUS              IMAGE
apps-polymarket-fetch-bot              Up XX               .../polymarket-fetch:latest
apps-polymarket-fetch-wallet-monitor   Up XX               .../polymarket-fetch:latest
apps-pm-arb-validate                   Up XX               .../pm-arb:latest

(apps-polymarket-fetch-scheduled-run is a timer-driven one-shot — its container only exists while the job is running, every 30 min.)

Inspect one app

APP=polymarket-fetch-bot
sudo systemctl status apps-$APP.service
sudo docker logs --tail 50 apps-$APP                # last 50 lines
sudo docker logs -f apps-$APP                       # follow live
sudo docker exec apps-$APP env | grep ^OTEL_        # confirm OTel auto-injection

For a kind: job:

APP=polymarket-fetch-scheduled-run
sudo systemctl list-timers apps-$APP.timer --no-pager   # next scheduled fire
sudo journalctl -u apps-$APP.service --since '1 hour ago' --no-pager

journalctl shows the systemd unit’s stdout (the docker compose run --rm app invocation’s output). For services that have logged a lot, prefer docker logs over journalctl — the latter shows ExecStart wrapping noise.

Bounce a single app

sudo systemctl restart apps-$APP.service

This re-runs ExecStartPre=docker pull then docker compose up -d. If the AR image’s :latest has been updated, you’ll get the new digest. The unit’s state: restarted behavior baked into the Ansible role is also systemctl restart-equivalent.

For a clean teardown without removing the unit:

sudo systemctl stop apps-$APP.service
sudo docker compose -f /opt/apps/$APP/docker-compose.yml down

(Stops the unit AND removes the container. start again brings it back.)

Inspect compose + env

Every app’s working directory:

ls /opt/apps/$APP/
# docker-compose.yml  .env

.env is rendered from secrets/<app>.env in the terraform repo with mode 0600 and ownership ops:ops. The compose file is templated by Ansible from ansible/roles/apps/templates/docker-compose.yml.j2.

Don’t hand-edit either — re-run make deploy APP=<name> (or gh workflow run deploy-apps.yml ... -f app=<name>) to re-render from source. Any edits get clobbered on next deploy.

OTel collector health

curl -s http://127.0.0.1:8888/metrics | grep -E '^otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)_(log_records|metric_points|spans)' | head -10

Three pairs to watch: logs/metrics/spans. accepted should equal sent. send_failed should be absent or 0. See Verifying signals are flowing for the canonical signal.

Tailscale on the VM

sudo systemctl status tailscaled
sudo tailscale status | head -5
sudo tailscale netcheck                      # UDP + DERP latency
sudo cat /etc/default/tailscaled              # shows the --debug=127.0.0.1:5252 flag for Prometheus metrics
curl -s http://127.0.0.1:5252/debug/metrics | grep -E '^(goroutines|magicsock_|derp_)' | head -10

If the tailnet hostname collides (see gcp-terraform-ansible-gotchas #15), check tailscale status from your laptop AND from the VM (sudo tailscale status). The VM’s own view shows it as <hostname> (no suffix); peers see whatever Tailscale assigned (may have -1).

Docker resource budget

free -h                                           # host RAM
sudo docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}'
df -h /

Reference for polymarket-infra (e2-small, 2GB):

StateUsed MEMAvailable
Idle (just collector + tailscale + dockerd)~350 MB~1.5 GiB
4 services running steady-state~650 MB~1.3 GiB
Scheduled-run firingspikes by ~150 MB for ~30sdips to ~1.1 GiB

available dropping below ~200 MB is when to consider e2-medium (4 GB) — see Resizing.

Common debug paths

”App is up but not doing anything”

  • docker logs --tail 100 apps-$APP — recent activity. Look for the startup brief, then steady-state work.
  • SigNoz logs view, filter service.name = $APP, sort by timestamp — see what the app thinks it’s doing.
  • Supabase MCP execute_sql to check whether DB writes are happening: SELECT count(*), MAX(<timestamp_col>) FROM pm_<table> for whichever table the app feeds.

”App keeps crash-looping”

  • sudo systemctl status apps-$APP.service shows the most recent exit code + restart count.
  • sudo docker logs --tail 50 apps-$APP shows the last error before crash.
  • Common: Supabase unreachable (IPv6/IPv4 issue — see gcp-terraform-ansible-gotchas #18), Tokio runtime panic (Rust app’s init order — see gcp-terraform-ansible-gotchas #21), or env var missing.

”Service shows active but container shows Restarting”

The systemd unit succeeded its ExecStart (the docker compose up -d call returned 0 even though the container then exited). Use docker logs and docker ps to see the actual container state. systemctl is-active is unreliable for compose-managed containers.

”Trace gap in SigNoz”

  • Was the service bounced recently? Check docker ps Status column for “Up X seconds”.
  • Did the OTel collector restart? sudo journalctl -u otelcol-contrib -n 50 --no-pager.
  • Did the network drop? sudo journalctl -u tailscaled -n 50 --no-pager | grep -iE 'derp|magicsock'.

Logs by service in SigNoz

The structured-logging conversion (gcp-terraform-ansible-gotchas #22) means logs are filterable by attribute. Useful queries:

GoalSigNoz logs query
All logs from one service in last hourservice.name = '<app_name>'
Errors onlyservice.name = '<app_name>' AND severity_text = 'ERROR'
Slow queriesseverity_text = 'WARN' AND body CONTAINS 'slow statement'
Single Telegram chat’s activityservice.name = 'polymarket-fetch-bot' AND chat_id = '-...'
pm-arb signals for one assetservice.name = 'pm-arb-validate' AND slug LIKE '%btc-updown%'

Restart EVERYTHING

make redeploy-all from operator laptop OR gh workflow run deploy-apps.yml ... -f app=all from anywhere. Bounces all kind: service apps. Timers re-fire at their next OnCalendar tick.

Graph View

Backlinks