A multi-agent re-validation (2026-06-01) found that the project’s headline edge numbers are PROVISIONAL, not proven: three production code paths leak future / post-entry information into the evidence that produced them. Until the signal is re-derived point-in-time and leakage-free, trust no current edge number.
Trust no current edge number until re-derived leakage-free
The two figures that have been treated as “the edge is real” are both built on contaminated evidence:
- the −44% copy-gate backtest ROI, and
- the short-term skill persistence result (Spearman 0.53, p = 0.025, “12 of 51 wallets with
edge_z > 2”). Three code paths leak lookahead / survivorship information into these. Treat both as unverified pending a leakage-free re-derivation.
For Agents
The three leaking code paths (verified by reading source, 2026-06-01):
crates/polymarket-data/src/consensus.rs:30(compute_consensus_groups) — aggregates LIVE positions (current_value/avg_price), i.e. post-entry, survivorship-biased state, not the as-of-trade state.crates/polymarket-fetch/src/backtest.rs(build_wallet_data) +crates/polymarket-data/src/metrics.rs:132(pair_trades_to_resolved) — pair trades to resolution payouts across the train/test cutoff → label leakage.crates/polymarket-fetch/src/repo.rs:1429(fetch_short_term_bet_rows) — has no time filter → survivorship + lookahead. The one sound piece to keep: the price-conditioned Monte-Carlo luck baseline inshort_term_accuracy.rs:259.
The three leaks
1. Consensus aggregates LIVE positions (post-entry + survivorship)
crates/polymarket-data/src/consensus.rs:30 — compute_consensus_groups aggregates live position fields (current_value, avg_price). These are the wallet’s position state now, not its state as of the trade. Two distinct biases ride in:
- Post-entry information —
current_valuealready reflects how the bet has played out since entry; it is not knowable at decision time. - Survivorship — wallets/positions that blew up or were redeemed are missing or rosier than they were, so the surviving set looks better than the real-time set did.
Any consensus signal built from this is a snapshot of winners-so-far, not a point-in-time “what are skilled wallets doing right now” signal.
2. Trade→resolution pairing crosses the train/test cutoff (label leakage)
crates/polymarket-fetch/src/backtest.rs (build_wallet_data) together with crates/polymarket-data/src/metrics.rs:132 (pair_trades_to_resolved) pair each trade to its resolution payout. The pairing is not bounded by the train/test cutoff — trades in the training window are labeled using resolutions that only became known in the test window (and vice versa). That is textbook label leakage: the model/score effectively peeks at the answer key. This is the most likely contaminant behind the −44% copy-gate ROI being treated as a clean out-of-sample number.
3. Short-term bet rows have no time filter (survivorship + lookahead)
crates/polymarket-fetch/src/repo.rs:1429 — fetch_short_term_bet_rows applies no time filter at all. Every analysis built on it (including the persistence figure: Spearman 0.53, p = 0.025, 12/51 wallets edge_z > 2) pulls the full lifetime row set, so:
- Survivorship — only wallets that are still present/active are in the set.
- Lookahead — rows from after any nominal cutoff are mixed in freely.
A persistence statistic computed over an unfiltered, survivorship-selected set is not measuring out-of-sample persistence.
The one sound piece to keep
Keep the price-conditioned Monte-Carlo luck baseline
short_term_accuracy.rs:259— the price-conditioned Monte-Carlo luck baseline (edge_luck_distribution, re-simulating each bet as a Bernoulli draw at the market price) is methodologically sound and should be kept and reused. It is the right null model; the problem is the data feeding the other statistics, not this baseline.
Remedy (López de Prado playbook)
To re-derive any edge number leakage-free:
- Reconstruct the signal point-in-time from
pm_trades— build each wallet/consensus feature from the state knowable as of the trade (trade_time), never from live position snapshots. - Add a real
resolved_attimestamp.pm_market_resolutionscurrently only hasfetched_at(when we learned the outcome), which is not when the market resolved — so payouts cannot currently be aligned to a true resolution time. A genuineresolved_atis a prerequisite for honest train/test separation. - Purged + embargoed CPCV (Combinatorial Purged Cross-Validation) — purge training observations whose labels overlap the test window and embargo a buffer after it, so trade→resolution overlap can’t leak across the cutoff.
- Negative-control shuffled-label sentinel — run the whole pipeline on shuffled labels; any “edge” that survives shuffling is leakage/artifact, not signal.
- Deflate by the Deflated Sharpe Ratio (DSR) — many strategy variants will be tried, so the best-looking one is inflated by multiple testing; deflate to get an honest significance.
Implication
Every current “the edge is real” claim — the −44% gated-copy backtest, the 0.53 / p=0.025 / 12-of-51 persistence result — is provisional. They may survive a clean re-derivation, but as of 2026-06-01 they are built on contaminated evidence and must not be cited as proven. This is the rigorous extension of the earlier no-edge finding in copy-trade-algorithm: not only does past-PnL not persist, but the very pipeline used to measure skill/persistence currently leaks future information.
Verified file:line references
| fact | location |
|---|---|
consensus aggregates LIVE positions (current_value/avg_price) — post-entry + survivorship | crates/polymarket-data/src/consensus.rs:30 (compute_consensus_groups) |
| trade→resolution pairing crosses train/test cutoff (label leakage) | crates/polymarket-fetch/src/backtest.rs (build_wallet_data) + crates/polymarket-data/src/metrics.rs:132 (pair_trades_to_resolved) |
| short-term bet rows have NO time filter (survivorship + lookahead) | crates/polymarket-fetch/src/repo.rs:1429 (fetch_short_term_bet_rows) |
| sound piece to keep — price-conditioned Monte-Carlo luck baseline | crates/polymarket-fetch/src/short_term_accuracy.rs:259 (edge_luck_distribution) |
pm_market_resolutions has only fetched_at, no true resolved_at | schema gap (remedy step 2) |
Related
- best-consensus-algo-design — the Phase 2+ algorithm this validation work must vet; its backtest is exactly the pipeline that needs the purged+embargoed CPCV / DSR / shuffled-label controls below
- copy-trade-algorithm — the no-edge backtest verdict this extends; the −44% gated-copy ROI is one of the contaminated numbers
- selector-vs-skill-gap — the rigorous skill layer (
short_term_accuracy.rs) whose persistence output (0.53 / p=0.025 / 12-of-51) is among the provisional figures - short-term-accuracy — home of the Monte-Carlo luck baseline that is the one sound piece to keep
- polymarket-fetch — workspace layout, schema (
pm_trades,pm_market_resolutions), backtest/metrics modules - TOPICS — project theme index