Levandor

Autotrade alert latency + scheduled-run timeouts (2026-06-17)

6 min read

Operational snapshot (2026-06-17, point-in-time) of why pmv2 autotrade buy/sell alerts arrive 20–38 min behind the source trade, plus a NEW finding: the scheduled-run job intermittently hits its 15-min systemd start timeout and is killed, dropping a whole cycle’s later-pipeline alerts. Facts gathered from journalctl on the polymarket-infra VM; extends the REST-poll lag already flagged as a strategic deferral in autotrade-3c2-complete-handoff.

Point-in-time snapshot + verification routing

All figures are a 2026-06-17 observation from journalctl/manifest on polymarket-infra. Run durations and the timeout counts are a moving target — re-verify before acting. Live VM facts here were gathered via direct read-only SSH; per operator policy, VM ops now route through @deploy on babylon — do NOT SSH the VM directly to re-check.

Production scheduler (current manifest)

Source: terraform manifest ~/levandor/terraform/deploys/apps.yml. This is the current (2026-06-17) topology and supersedes the older service table in polymarket-fetch-deploy (that note is the 2026-05-27 deploy record — wallet-monitor was still present then; it is now decommissioned).

UnitKindStateSchedule / restartNotes
polymarket-fetch-scheduled-runjobpresentschedule: "*:0/30" (every 30 min)cmd scheduled-run --dedup-seconds 300 --gap-warn-seconds 3600; image …/apps/polymarket-fetch:latest
polymarket-fetch-botservicepresentrestart: alwaysTelegram bot
polymarket-fetch-wallet-monitor—absentdecommissionedwas present at 2026-05-27 deploy
pm-arb-validate—absentdecommissioned
weather-bet-runservicepresentcontinuous
weather-bet-resolvejobpresentdaily 14:00

systemd unit type: Type=oneshot, TimeoutStartUSec=15min, RuntimeMaxUSec=infinity. The 15-min start timeout is the operative cap — a run that exceeds it is terminated mid-pipeline.

NEW finding — intermittent run timeouts

3 of 48 runs in 24h were KILLED at the 15-min start timeout

Over the last ~24h: 45 runs “Deactivated successfully” (normal) and 3 runs hit the TimeoutStartUSec=15min cap → systemd logged “start operation timed out. Terminating”, status=130/SIGINT and killed the process mid-run.

Timeouts at (UTC): Jun 16 20:00, Jun 17 08:30, Jun 17 09:00. The last two were consecutive (08:30 then 09:00) — a possible worsening trend; worth watching for a third in a row.

A killed run loses every later-pipeline stage that hadn’t run yet that cycle — exit/sell alerts especially, since they run after positions. Fire-and-forget tokio::spawn Telegram sends in flight near the kill are also lost.

Typical successful run = ~6–9 min (e.g. 08:00:13 → 08:08:01). So most runs finish well inside the cap; the timeouts are intermittent, not deterministic on data size.

Where the autotrade step sits in a run

The autotrade entry step runs ~7.5–8 min into each run — e.g. a BUY dry-run alert logged at 09:08:08 for a run that started 09:00:29. That late position is structural: execute_pipeline (scheduled_run.rs) runs run_pmv2_collect (50 traders × up to 20 trade pages) → resolutions sync → leaderboard before run_pmv2_positions (which is where autotrade ingests). Exit/sell alerts run after positions, i.e. even later in the run — which is exactly the slice a 15-min kill tends to drop.

Latency decomposition — why alerts are so late

For Agents — autotrade alert latency budget

Net: buy alerts typically arrive ~20–25 min behind the source trade, up to ~38 min, occasionally a whole cycle skipped. Telegram delivery itself is sub-second and is NOT the cause. The latency is three stacked components:

  1. Detection lag (REST poll on the 30-min timer) → mean ~15 min, up to 30 min. The first_mover_core trigger is REST-polled, not WSS. (Same root deferral as autotrade-3c2-complete-handoff / autotrade-sources-signal-emitter.)
  2. In-run delay (~8 min) before the autotrade step executes — collect + resolutions + leaderboard run first in execute_pipeline.
  3. Intermittent whole-cycle loss — when a run is killed at 15 min, later stages (exit/sell alerts run after positions) are dropped for that cycle, and in-flight fire-and-forget Telegram sends near the kill are lost.

Autotrade is live in DRY-RUN in prod (mode=off / dry): observed 🧪 DRY-RUN — would BUY … notifications with bet links and via 0x… · 0x… attribution lines. So these alerts are the dry-run rehearsal stream, not live fills.

Candidate causes of the 15-min timeout (hypotheses — NOT confirmed)

Most likely: an occasional silently-hanging proxied HTTP call

The timeouts are intermittent rather than data-size-deterministic, which points at a hang, not slow bulk work. Known gap: a proxied reqwest client without connect_timeout can hang silently (zero warnings) — see the reference_proxied-client-connect-timeout memory. That failure mode matches an occasional whole-run stall.

Where the hang is: the 09:00 run got past the autotrade step (BUY dry-run logged 09:08) and was killed at 09:15 → the stall is somewhere in the later pipeline stages: trending alert / exit alert / category sync / paper copies / settlement / self sync / resolution check / overlap watcher / self-trade-changes.

Secondary contributor: the growing EXITED-events backlog. Observed slow query (exit-events cursor read):

SELECT proxy_wallet, condition_id, outcome_index, event_slug, slug, detected_ts
FROM pmv2_position_events
WHERE kind = 'EXITED' AND ($1::timestamptz IS NULL OR detected_ts > $1)
ORDER BY detected_ts ASC

returned 37,741 rows in 2.87s — a large/growing backlog suggesting the cursor may be near-NULL / not advancing. On its own 2.87s doesn’t blow a 15-min budget, but it compounds and will worsen as the backlog grows.

Suggested follow-ups (not yet done)

  • Add connect_timeout (not just timeout) to every proxied reqwest client in the later pipeline stages; bound fan-out with per-item tokio::timeout. (The known silent-hang fix.)
  • Watch for a 3rd consecutive timeout — 08:30 + 09:00 were back-to-back.
  • Investigate why the autotrade_exits cursor isn’t advancing past a 37k-row EXITED backlog (cursor near-NULL).
  • Strategic: a real-time on-chain OrderFilled WSS trigger removes component (1) of the latency budget and is already recommended before scaling past tiny caps (autotrade-3c2-complete-handoff).
  • Re-verify all of the above via @deploy on babylon, not direct SSH.

Graph View

Backlinks