Levandor

Watcher On-Chain Gap-Backfill Sweep (eth_getLogs reconciliation)

5 min read

The watcher’s Polygon WSS log feed (run_log_feed) auto-reconnects, but eth_subscribe NEVER replays logs emitted during the disconnect window — so cohort / self / our-fill OrderFilled logs in the gap were lost permanently. Fix = a 30s periodic eth_getLogs reconciliation sweep over [last_seen_block, latest] that re-fetches the missed logs and feeds each one through the IDENTICAL worker path as a live WatchEvent. Shipped to main as commit 43023ba (2026-06-20); gate green (563 polymarket-fetch + 107 polymarket-data); NOT yet redeployed.

For Agents — state at a glance

  • New file: crates/polymarket-fetch/src/pmv2/watch/backfill.rs (the sweep).
  • New public primitives in crates/polymarket-data/src/onchain.rs: eth_block_number, eth_get_order_filled_logs, cohort_address_topics, build_get_logs_filter (modeled on the existing eth_call_u128 HTTP-RPC glue).
  • Watermark: last_seen_block: Arc<AtomicU64>, advanced via fetch_max inside handle_order_filled right after decode.
  • Uses the existing POLYGON_RPC_URL HTTP endpoint (NOT the WSS feed) for the sweep.
  • The whole thing is provably safe because every downstream consumer is ALREADY idempotent (see Safety). Live/backfill overlap is harmless by construction.
  • Plan doc: docs/superpowers/plans/2026-06-20-pmv2-watcher-gap-backfill.md.

The Gap

The feed already handles disconnects gracefully — run_log_feed (in polymarket-data/src/onchain.rs, the Polygon WSS eth_subscribe logs substrate, see On-Chain Detection) reconnects with backoff. But reconnection only resubscribes for new logs going forward. The JSON-RPC eth_subscribe has no replay: any OrderFilled log emitted between the drop and the resubscribe is gone for good.

That window silently drops:

  • cohort OrderFilled logs → a first-mover entry/exit never detected,
  • self fills → self-trade alerts missed,
  • our-fill logs → autotrade reconcile loses a fill confirmation.

Every gap cause matters here, not just clean disconnects: a silent frame drop or a provider hiccup leaves the socket “up” while logs vanish.

Solution — periodic eth_getLogs sweep

backfill.rs runs every 30s. Each pass:

  1. latest = eth_block_number().
  2. from = last_seen_block watermark (see below) — INCLUSIVE.
  3. eth_get_order_filled_logs(from, latest, cohort) — re-fetches OrderFilled logs for the cohort across both the maker and taker topic slots, mirroring the live feed’s build_logs_payload (via cohort_address_topics + build_get_logs_filter).
  4. Each recovered log is fed through the IDENTICAL worker path as a live WatchEvent (try_send into the same worker queue).

Why a periodic sweep, NOT a literal on-reconnect hook

  • Catches ALL gap causes — disconnect, silent frame drop, provider hiccup — not just full socket disconnects. An on-reconnect hook only fires on the cases the client actually noticed.
  • Lives entirely in the watcher. No surgery on the generic polymarket-data feed crate (which is shared substrate).
  • Simpler to test — a plain periodic loop over a watermark, no entanglement with the reconnect state machine.

Safety — the key insight

Feeding backfilled logs is provably safe because every downstream consumer is already idempotent. Live and backfill can deliver the same log and nothing double-acts:

ConsumerIdempotency mechanism
Consensus (2nd-sharp) alertfires on the 2nd distinct holder by WALLET — a re-fed log for an already-counted wallet is a no-op (see tracked-alerts-consumer)
First-mover entry (autotrade)engine dedups at ON CONFLICT(dedup_key) (see cohort-firstmover-nats-migration)
Self-trade alertDB dedup by (wallet, tx, outcome)
Our-fill reconcileshares the worker’s seen set keyed by (tx_hash, log_index)

So live/backfill overlap is harmless — it’s structurally equivalent to the pre-existing first-mover match-cron backstop (the ~30-min REST sweep that already re-derives the same entries). The sweep just closes the gap faster and over a different transport.

Mechanics

  • Watermark. last_seen_block: Arc<AtomicU64>, advanced by fetch_max inside handle_order_filled immediately after a successful decode. fetch_max (not a plain store) means out-of-order delivery can never rewind the watermark.
  • Inclusive from. from = watermark (NOT watermark + 1) is deliberate: it recovers intra-block partial-delivery gaps (the feed delivered some logs of a block then dropped). The re-fetch overlaps the last block, and the consumers’ dedup absorbs it.
  • Cap. MAX_BACKFILL_BLOCKS = 2000 bounds the [from, latest] span; a log line fires when the cap is hit (no silent truncation — you can see when a gap exceeded the window).
  • Non-blocking. try_send into the worker queue, so the sweep can never block the live worker if the queue is full.
  • Reuses HTTP RPC. Same POLYGON_RPC_URL HTTP endpoint as eth_call_u128 — the sweep does NOT touch the WSS feed.

Inclusive from + cap interaction

The inclusive from means the sweep re-fetches the watermark block every pass — cheap and dedup-absorbed, but be aware it is intentional, not an off-by-one. And MAX_BACKFILL_BLOCKS = 2000: a disconnect longer than ~2000 Polygon blocks (~1 hr at ~2s/block) is only partially recovered by one sweep — the watermark advances and subsequent sweeps catch up, but watch the cap-hit log if a long outage is suspected. The first-mover match-cron is still the deeper backstop.

Deferred (noted, not done)

  • Universal (tx_hash, log_index) front-dedup. Unnecessary — every consumer is already idempotent. Adding a front-dedup would also alter the subtle both-parties-in-cohort double-delivery behavior (when both maker and taker are cohort wallets, the same tx legitimately surfaces twice), so it was deliberately left out.
  • Instant on-reconnect replay. The sweep recovers within ≤30s; a dedicated reconnect-triggered replay was judged not worth the coupling.
  • Deep reorg handling. Out of scope — this is a detection signal pipeline, not settlement. A reorg that rewrites a recovered log is a settlement-layer concern.

Graph View

Backlinks