Levandor

pmv2 Pipeline Bug Audit & Fixes (2026-06-02)

7 min read

A 4-reviewer pass over the full pmv2 pipeline (collect/score → positions/watcher → resolve → orchestration) done before enabling the first-mover watcher. ~40 raw findings; most “Critical/High” were over-claimed and triaged out. One real latent bug was fixed in the LIVE pipeline (a missing dedup before a chunked ON CONFLICT DO UPDATE — the exact crash class that killed scheduled-run #371). Three watcher-hardening fixes landed in the still-default-OFF, unreleased watcher. A sharp architectural constraint was surfaced: winner-selection logic is SHARED between resolution PnL and the validated trader-scoring algo and MUST NOT be changed casually.

For Agents

Read this before touching pmv2 resolution / positions / watcher code. Key takeaways:

  • GOTCHA (durable): ANY chunked ON CONFLICT (...) DO UPDATE upsert in this codebase MUST dedup its input by the conflict key first, or it crashes with ON CONFLICT DO UPDATE command cannot affect row a second time when one chunk contains two rows with the same conflict key. This already took down scheduled-run #371 (positions table) and was latent again in upsert_market_resolutions.
  • CONSTRAINT (sharp edge): repo::load_event_winners is SHARED — it feeds BOTH the watcher resolution/PnL pass (positions.rs) AND the validated edge/consistency trader scoring (mod.rs → compute_trader_metrics). Changing winner-selection logic (50/50 payout-DESC tiebreak, NaN-drop guard) shifts every trader’s edge and breaks the validated pmv2-two-tier-trader-scoring algo. Any resolution-winner change requires re-validating scoring.
  • State: all fixes are behavior-neutral to scoring; gates green (164 tests, clippy clean except 3 pre-existing, fmt clean); NOT YET committed; watcher migration 20260602030000_pmv2_watcher.sql edited (added UNIQUE) but NOT applied.

The audit

Four review lenses over the four pmv2 pipeline stages:

  1. collect / score (pmv2/mod.rs, pmv2/scoring.rs)
  2. positions / watcher (pmv2/positions.rs)
  3. resolve (pmv2/repo.rs resolution path)
  4. orchestration (scheduled-run wiring)

~40 raw findings surfaced. Most flagged “Critical” / “High” were over-claimed and triaged out (see Over-claimed — NOT bugs). Net: 1 real latent live bug, 3 watcher-hardening fixes, 1 architectural constraint discovered, several deferred items.

The one real latent bug (LIVE pipeline)

Missing dedup before chunked ON CONFLICT DO UPDATE

upsert_market_resolutions / flatten_resolution_rows in crates/polymarket-fetch/src/pmv2/repo.rs had NO dedup before an ON CONFLICT (condition_id, outcome_index) DO UPDATE chunked insert. Two rows with the same (condition_id, outcome_index) in one chunk → ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time. This is the EXACT same crash class that took down scheduled-run #371 (which was the positions table, not resolutions). Latent here because the resolution feed can legitimately surface the same key twice in one batch.

Fix

Added dedup_resolution_rows — keep-last, order-preserving — applied before chunking. Keep-last is behavior-identical to what DO UPDATE converges to (the last write wins), so this is a pure crash-prevention fix with zero behavior change.

Durable lesson

Any chunked DO UPDATE upsert in this codebase must dedup its input by the conflict key first. INSERT ... ON CONFLICT DO UPDATE cannot touch the same target row twice within a single statement — chunked batches make this a live hazard whenever the source can repeat a key. This is now the second occurrence of this exact class (positions = #371, resolutions = this audit).

Architectural constraint discovered

load_event_winners is SHARED — do not "fix" winner-selection casually

repo::load_event_winners feeds two consumers:

  1. the watcher resolution / PnL pass (pmv2/positions.rs), and
  2. the validated edge/consistency trader scoring (pmv2/mod.rs → compute_trader_metrics).

Therefore the winner-selection logic — specifically the payout-DESC tiebreak on 50/50 markets and the NaN-drop guard — MUST NOT be changed casually. Altering it shifts every trader’s edge, which changes the Tier-1 trader_score, which breaks the validated pmv2-two-tier-trader-scoring algo. Any resolution-winner change requires re-validating scoring end-to-end. Treat this function as a frozen interface unless you’re explicitly re-running the scoring validation.

Three watcher-hardening fixes (watcher is default-OFF, unreleased)

All in crates/polymarket-fetch/src/pmv2/positions.rs + the watcher migration. The watcher is not yet released (default-OFF), so these are pre-release hardening, not live-incident fixes.

1. load_copy_config could error the whole positions snapshot

load_copy_config was a fetch_one called unconditionally before the enabled-check. A missing singleton config row would error the ENTIRE positions snapshot, not just the watcher.

Fix → fetch_optional + a CopyConfig::disabled() default when the row is absent. A missing watcher config now degrades to “watcher off,” leaving the core snapshot intact.

2. insert_position_event swallowed errors + no unique constraint

insert_position_event swallowed errors via .ok(), and the table had no unique constraint → silent audit-log loss and duplicate ENTERED rows on overlapping runs.

Fix →

  • tracing::warn on insert error (no more silent loss),
  • UNIQUE(proxy_wallet, condition_id, outcome_index) on the table,
  • ON CONFLICT DO NOTHING on the insert.

Migration not applied

The UNIQUE was added to supabase/migrations/20260602030000_pmv2_watcher.sql, which is edited but NOT applied. Apply it before enabling the watcher.

3. Per-event cap was bypassable within a single cycle

The per-event cap was bypassable inside one run: event_cap_used read only committed DB rows, so N new entries for the same event_id in one cycle each saw used=0 and all inserted, blowing past per_event_cap_usd.

Fix →

  • an in-flight HashMap<event_id, f64> accumulator tracking entries decided within the current cycle,
  • a pure within_event_cap helper (testable, no DB),
  • added AND status='open' to the event_cap_used query so the cap = live exposure (resolved copies no longer count against the cap).

Over-claimed — NOT bugs

Recorded so these are not re-investigated next session. The scoring math in pmv2/scoring.rs (and model) is sound:

  • “n=1 → consistency=0” — unreachable: the min_resolved=50 floor gates it out before consistency is ever computed on n=1.
  • “consistency_factor deviates from min(early, recent)” — wrong: the clamp(min / max(overall, 0.01), 0, 1) ratio form is correct BY DESIGN, so that long_term = consistency × span stays in [0,1], keeping the score multiplier in [0.5, 1.0]. It is intentionally a ratio, not a raw min.
  • “negative-edge → negative-score is a bug” — intentional: negative scores rank losers last (the whole point of edge-based ranking; see pmv2-two-tier-trader-scoring).
  • “NaN can poison the sort” — impossible: scores are stored as NUMERIC(6,5); NaN cannot occur in the sort key.

Deferred (real, but out of scope / would change behavior or need tuning)

NOT fixed this pass. Each is genuine but either needs tuning data, a separate resilience pass, or would alter behavior:

  • Sizing-formula tuning — score_weight = score·10 + 1 makes base_usd a near-no-op above score ~0.4 (everything clamps to max_size_usd). Tune at enable-time with the real cohort.
  • No per-wallet tokio timeout on fetch_candidate_trades — only the reqwest-level timeout exists. A resilience pass item (same class as the connect_timeout lesson in pmv2-two-tier-trader-scoring).
  • No min-success floor — a mass-timeout run completes with a silently degraded cohort (no “too few wallets succeeded → abort” guard).
  • Multiple DB pools per pipeline run — pressures the Supabase connection budget; consolidate to a shared pool.
  • Overlapping 30-min runs — runs take 11–23 min; the dedup guard prevents simultaneous starts only, not a slow run overlapping the next tick.

State

  • All fixes are behavior-neutral to scoring.
  • Gates green: 164 tests pass, clippy clean (except 3 pre-existing nits), fmt clean.
  • NOT YET committed.
  • Watcher migration supabase/migrations/20260602030000_pmv2_watcher.sql edited (added UNIQUE) but NOT applied.

Graph View

Backlinks