Levandor

Agent Guide — Remove an App

4 min read

For LLM agents

Removing an app is a TWO-step ritual: first flip state: absent so the role tears down systemd + container + app dir cleanly, THEN delete the manifest entry. Don’t shortcut by deleting the entry directly — that leaves orphaned systemd units and /opt/apps/<name>/ on the VM forever.

The two-step

Step 1 — set state: absent, deploy, verify

Edit /Users/levander/levandor/terraform/deploys/apps.yml. Find the entry, change state: present to state: absent. Leave the rest of the entry intact — the role needs name, kind, image, env_file to construct the unit paths it’s tearing down.

- name: <app_name>
  kind: service                # keep
  image: <image>               # keep
  env_file: secrets/<name>.env # keep
  state: absent                # CHANGED from present
  compose_extra:               # keep
    ...

Commit + push (so CI has the change), then redeploy:

gh workflow run deploy-apps.yml -R wowjeeez/terraform --ref main -f app=<app_name>

Or laptop-side: cd /Users/levander/levandor/terraform && make deploy APP=<app_name>.

The role dispatches to _absent.yml, which:

  1. Stops the systemd service (failed_when: false — succeeds even if already stopped).
  2. Disables it.
  3. For kind: job: stops + disables the .timer too.
  4. Removes /etc/systemd/system/apps-<name>.service and apps-<name>.timer.
  5. Removes /opt/apps/<name>/ recursively (compose file + .env + nothing else by convention).
  6. Notifies daemon-reload.

Verify:

ssh ops@polymarket-infra '
  echo "===unit status (should report not-found)==="
  sudo systemctl status apps-<name>.service 2>&1 | head -3
  echo "===app dir (should not exist)==="
  ls -la /opt/apps/<name> 2>&1
  echo "===container (should not exist)==="
  sudo docker ps -a --filter "name=apps-<name>" --format "{{.Names}}: {{.Status}}"
'

Expected output: “Unit apps-.service could not be found”, “No such file or directory” on the app dir, no container row.

Step 2 — delete the manifest entry

Once Step 1 verifies clean, edit deploys/apps.yml again and remove the entire entry. Commit + push.

You can also remove secrets/<name>.env if no other entry uses it (some apps share an env file — check grep env_file deploys/apps.yml before deleting).

If the entry was the only one in the manifest, you can leave a placeholder apps: [] — the role short-circuits cleanly on an empty list.

Why two-step (not one)

If you delete the manifest entry without first running state: absent:

  • The role iterates only the entries still present. The removed app is invisible to it.
  • The systemd unit files stay on the VM (/etc/systemd/system/apps-<name>.service).
  • The container keeps running (no one told it to stop).
  • /opt/apps/<name>/ keeps holding the env file and compose file.
  • On make redeploy-all, the orphan unit still tries to docker pull on every restart — quiet waste of bandwidth.

The first cleanup attempt then requires either manual ssh ops@polymarket-infra 'sudo rm /etc/systemd/system/apps-<name>.* && sudo systemctl daemon-reload && sudo systemctl stop apps-<name>' per app, or reverting your manifest delete, adding state: absent, and redeploying.

Don't try to clean up by hand

Manual rm/systemctl stop works but the next make redeploy-all is the source of truth for what should exist — if the manifest still references the app, the role re-creates everything. Always go through the manifest.

Edge cases

Removing a kind: job mid-cycle

If the job’s timer is about to fire when you redeploy with state: absent, the role stops the timer first, so the next firing never happens. Any currently-running instance of the job’s one-shot container completes its current iteration and exits normally (the role does not kill in-flight containers). Acceptable for idempotent jobs (like polymarket-fetch scheduled-run --dedup-seconds 300). For non-idempotent jobs, manually docker stop apps-<name> before flipping the state if you need a hard cut.

Removing the LAST app

If you set the LAST entry’s state: absent and remove it, deploys/apps.yml becomes apps: []. The role handles this — it just iterates an empty list. No special handling needed.

Removing an app that’s referenced by another app’s env file

secrets/<name>.env is just a per-app file. If multiple apps share one (e.g., all three polymarket-fetch services share secrets/polymarket-fetch.env), removing one app doesn’t touch the env file. Manually delete the env file only when nothing references it.

”I want the data but not the container”

For apps that wrote to a host bind-mount (e.g., pm-arb-validate → /var/lib/pm-arb-paper/, wallet-monitor → /var/lib/wallet-monitor/), the bind-mount source on the host is NOT removed by state: absent (the role only manages /opt/apps/<name>/). To keep the JSONLs after removal: do nothing extra. To wipe them: sudo rm -rf /var/lib/<name> over SSH after Step 1.

Graph View

Backlinks