Levandor

Cohort consensus backtest — flat €10, 2 weeks (2026-07-04)

6 min read

Counterfactual replay of the production consensus signal (fire on the 4th distinct cohort wallet to BUY a market) over the past 2 weeks with flat €10 bets — asking “what P&L would we have booked had we followed it?“. Point-in-time snapshot, not a validated strategy result.

For Agents

Verdict (this window only): 98 consensus fires → 93 resolved → 53W/40L (56.99% hit-rate) → +€113.44 realized, +12.20% ROI on €930 staked; +€144.74 / +14.77% incl. 5 open positions marked-to-market. Robustness: LOW. n=93 resolved, 14-day window, heavily event-clustered (World Cup + Wimbledon). Genuine-edge signal over this window, but not statistically robust and not a purged/embargoed harness — do not treat as a shippable-strategy number. No backtest harness exists in the repo. This was a standalone Python replay against live Polymarket APIs. Artifacts (consensus_backtest.py, consensus_signals.json, consensus_summary.json) live in an ephemeral session scratchpad — not committed, not durable. This note is the durable record. €-figures are USDC at a flat 1 USDC ≈ €1 assumption — there is no EUR/FX handling anywhere in the codebase.

What was tested

“What if we had followed the production consensus signal (the 4 top-trader converge alert) over the past 2 weeks with flat €10 bets.” This replays the entry side of the live consensus alert as if each fire were auto-followed with a fixed stake, then settles each position against real Polymarket resolutions.

This is the auto-copy counterfactual of the human-ALERT consensus gate documented in tracked-alerts-consumer — distinct from (and complementary to) the backtested “≥K agree” auto-copy thesis that xref-consensus-rejected-fresh-copy-wins rejected. See Reconciling with the xref result below.

Methodology (faithful to production)

  • Fire logic — replayed the exact ConsensusTracker::record_buy from crates/cohort_algo/src/pmv2/tracked/alerts_consumer.rs:100-116: per (condition_id, outcome_index), a rolling 48h window (CONSENSUS_WINDOW=172800s); fire when the 4th DISTINCT cohort wallet BUYs (is_new && holders.len()==4, CONSENSUS_MIN_HOLDERS=4). BUY-side only — SELLs are ignored for entry. (The 4-holder threshold is the current production value; it was raised from 2→4 on 2026-06-25 per cohort-firstmover-roll-d851d44-2026-06-25, so this replay matches live prod, not the older 2-holder gate.)
  • Cohort — the 14 core traders with score>0 from the latest scored run (run_id 1991, pmv2_traders) — the exact set the live watcher arms on. Selection provenance: pmv2-two-tier-trader-scoring.
  • Window — 2026-06-20 18:33 → 2026-07-04 18:33 UTC (14 days).
  • Entry — entry price = the triggering (4th) buy’s fill price. Flat €10 stake ⇒ shares = 10 / p. 1 USDC ≈ €1.
  • Settlement — via gamma /markets: resolved iff closed==true, winner = the outcome whose outcomePrices[i] >= 0.5 (mirrors parse_market_resolution in crates/polymarket-data/src/client.rs). Open markets marked-to-market at current price.

Results (verified, real run)

  • 98 consensus fires, all 98 distinct (condition_id, outcome_index) markets — 0 markets re-fired.
  • Resolution: 93 resolved · 5 still open · 0 unknown.
BucketMarketsStakedP&LROI
Resolved93 (53W / 40L)€930+€113.44+12.20%
Open (marked-to-market)5€50+€31.30—
Combined98€980+€144.74+14.77%
  • Resolved hit-rate 56.99% (53/93). The 40 losers are −€10 each.
  • Biggest winners: Detroit Tigers ML @0.19 (+€42.63), Germany “No” @0.25 (+€30.05), Netherlands “No” @0.26 (+€28.46), Brazil @0.29 (+€24.48).
  • Market mix: 87 soccer / World-Cup-type · 9 tennis (Wimbledon) · 2 baseball. Median entry price 0.49.
  • The 5 open positions’ +€31.30 is mark-to-market and moves run-to-run.

Interpretation

A 57% win rate at ~0.49 median entry producing +12% realized ROI implies the consensus signal carried genuine edge over this window — winning slightly more than half of near-coin-flip-priced bets is the shape of real edge.

But treat as point-in-time, not validated:

  • Small sample — only 93 resolved.
  • Short window — 2 weeks.
  • Heavily event-clustered — dominated by one World Cup cycle (+ Wimbledon); outcomes are correlated, so the effective independent-sample count is far below 93.
  • Not a purged/embargoed walk-forward — does not clear the purged-backtest-gate-no-edge ship bar (beat ~+2.75% random under the leakage-clean harness) and was never run through it.

Reconciling with the xref result

xref-consensus-rejected-fresh-copy-wins found consensus is structurally late on a leakage-clean purged harness (K1 first-mover +7.59% ≫ K2 worse ≫ K3/K4 ≈ noise/negative across 34,957 fires / 3 folds). The live signal here fires at the 4th converging buyer — effectively a K4 co-buy gate — yet booked +12% on this window. This does not overturn xref:

  • Entry-price bias — this replay enters at the 4th buyer’s own fill price (the cohort’s favorable price), the same upper-bound bias xref flagged; the live system would enter at the current best-ask (worse) instead.
  • Sample — 93 correlated, event-clustered resolutions here vs 34,957 fires across 3 disjoint purged folds there.
  • No leakage controls — point-in-time settlement, but no purge/embargo/negative-control.

So the +12% is best read as a favorable point-in-time snapshot at an upper-bound entry price, consistent with xref’s “trust the ordering, not the magnitude” caveat — not evidence that late consensus beats first-mover.

Key caveats / data completeness

Data-completeness — fires reported are a LOWER BOUND

The data-api /trades endpoint ignores startTime server-side and caps offset (only ~3–4k newest records reachable). 4 of the 14 cohort wallets were hyper-active (>4000 buys in-window) and hit the cap, so their oldest in-window buys are unreachable. This can only suppress early-window fires, never fabricate them ⇒ 98 fires is a lower bound, and every reported fire and P&L figure is accurate. This is the same /trades offset ceiling that bites trader scoring — see polymarket-data-api-trades-quirks (to consolidate), currently documented under pmv2-two-tier-trader-scoring (the ~3500/wallet ceiling; --trades-max-pages does not lift it) and polymarket-fetch.

Modeling choices that would change the number:

  • Entry price — replay uses the 4th buyer’s fill price; the live system enters at the current best-ask at detection (worse, and latency-exposed).
  • Exit policy — replay holds to resolution; the production exit policy (follow = auto-sell vs notify = alert-only) is deferred and unwired — see exit-mode-follow-notify-deferred. Real P&L would depend on whichever exit mode ships.
  • Sizing — replay uses flat €10; production uses conviction / score-weighted sizing (signals/sizing.rs).

Graph View

Backlinks