The rigorous luck/skill statistics this project computes (Wilson lower bound, edge-over-price, Monte-Carlo luck baseline, split-half persistence) live in a measurement layer that is not wired into the production wallet selector. The default selector ranks wallets purely by cumulative PnL. So the cohort/consensus signal is chosen by PnL-leaderboard intersection, while the genuinely defensible skill metrics sit unused in a separate table and CLI. (Verified by reading the code directly, 2026-06-01.)
For Agents
Two layers that should be connected but aren’t:
- Selection layer (what picks the cohort):
crates/polymarket-fetch/src/selector.rs. Default =top_n_intersection, ranks by cumulative PnL only. No statistical luck control.- Skill-measurement layer (rigorous, unused for selection):
crates/polymarket-fetch/src/short_term_accuracy.rs→pm_short_term_trader_accuracytable, surfaced by theshort-term-accuracy/ranked-traders/diagnose-persistencecommands. The fix is a selector that draws frompm_short_term_trader_accuracy(n_markets >= 40ANDedge_lower > 0) instead of the PnL leaderboard.
The default selector ranks purely by PnL
The production default is top_n_intersection (crates/polymarket-fetch/src/selector.rs:183, the TopNIntersection arm of Selector::select), which delegates to polymarket_data::select_top_n_intersection (defined in crates/polymarket-data/src/pipeline.rs:253). It:
- Takes the top-50 by PnL in each of the WEEK / MONTH / ALL leaderboards.
- Keeps wallets present in
>= min_windowsleaderboards (default 3). - Ranks and filters purely by cumulative PnL.
The only “consistency” control is the 3-window presence gate. There is no statistical luck control — no sample-size correction, no volume normalization, no edge-vs-price test. A wallet that got lucky on a few large longshots and a wallet with a real repeatable edge are indistinguishable to this selector as long as both clear the PnL leaderboards in 3 windows.
Concrete consequence — the consensus signal can rest on 2 PnL-whales
On the fresh OVERALL run 803 (2026-06-01),
top_n_intersectionyieldedintersection_size = 2wallets. The cohort/consensus signal currently rests on two PnL-whales — not on a skill-screened set. That is the entire basis of the “what reputable wallets are doing” signal under the default selector.
The rigorous skill layer exists — but isn’t in the selection path
crates/polymarket-fetch/src/short_term_accuracy.rs is a genuinely rigorous skill/luck-measurement layer. It is not consulted by any selector. Per wallet it computes:
- Wilson lower bound on hit-rate (
Z = 1.96,wilson_lower_boundatshort_term_accuracy.rs:63). Explicitly penalizes small samples — pinned by the testedge_lower_punishes_single_lucky_win(:524). - edge =
hit_rate − avg_entry_price. - edge_lower (conservative) =
wilson_lower − avg_entry_price. - edge_z =
edge / standard_error. - Brier score (calibration).
- split-half persistence (
split_halves,:216) gated atPERSISTENCE_MIN_MARKETS = 40markets (:11,:217,:294). - Monte-Carlo luck baseline (
edge_luck_distribution,LUCK_ITERATIONS = 200,:12,:259/:264/:265): re-simulates every bet as a Bernoulli draw at the market price, reportingluck_p_valueandsigmas_above_null(:319,:335–:336).
Results land in the pm_short_term_trader_accuracy table and are surfaced by the short-term-accuracy / ranked-traders / diagnose-persistence commands — not in the selection path that builds the watched cohort.
consistently_profitable is a partial improvement, not the fix
The consistently_profitable selector (select_consistently_profitable, crates/polymarket-fetch/src/selector.rs:342) is a step up from raw intersection — it uses rolling leaderboard appearances plus avg PnL / volume and ROI / composite thresholds (reads from pm_leaderboard_snapshots). But it still filters on PnL / volume, not on edge / Wilson / persistence, and it is not the default. So even the better-of-the-two selectors does not consult the skill-measurement layer.
Recommended fix — connect the two layers
Add a selector that draws from pm_short_term_trader_accuracy rather than from the PnL leaderboard intersection:
n_markets >= 40(thePERSISTENCE_MIN_MARKETSsample floor already enforced in the accuracy layer).edge_lower > 0(positive conservative edge over price).- optionally an
edge_zfloor (e.g. require statistically separated-from-zero edge).
This is the single change that makes selection skill-based instead of PnL-based: it wires the good measurement layer into the layer that actually decides which wallets the system watches and treats as consensus.
Why this matters — ties into the no-edge backtest finding
This gap is the structural reason the earlier negative result keeps biting. In copy-trade-algorithm the backtest showed that ranking wallets by top-cumulative-PnL has persistence ~0.21 Spearman, which did not beat the luck baseline ~0.23 — past PnL just doesn’t carry forward. That is exactly what the default top_n_intersection selector is doing: selecting on the metric (PnL) that was shown not to persist. It is why consensus was demoted to “information, not instructions.” Selecting on edge_lower / persistence instead of PnL is the principled response to that finding — the measurement layer that produced the luck-baseline verdict is the same one that should drive selection.
Verified file:line references
| fact | location |
|---|---|
default selector = top_n_intersection, delegates to polymarket_data | crates/polymarket-fetch/src/selector.rs:183 → crates/polymarket-data/src/pipeline.rs:253 |
consistently_profitable selector (PnL/volume, not default) | crates/polymarket-fetch/src/selector.rs:342 |
Wilson lower bound (Z = 1.96) | crates/polymarket-fetch/src/short_term_accuracy.rs:63 |
| small-sample penalty pinned by test | short_term_accuracy.rs:524 (edge_lower_punishes_single_lucky_win) |
PERSISTENCE_MIN_MARKETS = 40, persistence gate | short_term_accuracy.rs:11, :217, :294 |
| split-half persistence | short_term_accuracy.rs:216 (split_halves) |
Monte-Carlo luck baseline, LUCK_ITERATIONS = 200 | short_term_accuracy.rs:12, :259/:264/:265 (edge_luck_distribution) |
luck_p_value, sigmas_above_null | short_term_accuracy.rs:319, :335–:336 |
Related
- short-term-accuracy — the skill-measurement layer itself (Wilson, edge, edge_z, persistence) that this note argues should drive selection
- copy-trade-algorithm — the backtest showing top-by-PnL persistence (~0.21) did not beat the luck baseline (~0.23); the reason selecting on PnL is the wrong target
- top-trader-edge — the real per-wallet edge (
edge_z > 2) that an edge-based selector would surface - polymarket-fetch — Selector trait, the 4 selector impls, schema, CLI
- TOPICS — project theme index