The leakage-clean re-derivation called for in consensus-validation-leakage is done, and the verdict is negative: under a purged + embargoed walk-forward harness whose negative control reads ≈0, the production PnL/copy_score recommendation gate has NO edge — it returns −2.78% and loses to every naive baseline, including random. The old −43.37% gate ROI that the leakage doc flagged as provisional is now confirmed to have been a leakage artifact. Net: the current recommendation is mildly anti-selective (it picks worse-than-random trades).
For Agents
Purged + embargoed walk-forward, 45-day cutoff, 200 bps slippage, 3 disjoint 15-day folds. Pooled ROI by arm:
arm pooled ROI gate: recommended −2.78% baseline: all watched +3.68% baseline: random +2.75% baseline: raw-pnl top-20 (purged) −3.24% negative-control (shuffled labels) −0.04% (≈0 → leakage-clean) old (leaky / unpurged) gate −43.37% (LEAKAGE ARTIFACT) Relative ordering: gate (−2.78%) < random (+2.75%) < all-watched (+3.68%). The gate loses to the baselines in 2 of 3 folds. Phase 2 bar: any new signal must beat ~+2.75% random under this same purged harness to be worth shipping.
Verdict — the gate is anti-selective
Under a leakage-clean harness, the production PnL/copy_score recommendation gate:
- returns −2.78% pooled ROI,
- underperforms doing nothing fancy — it loses to all watched (+3.68%), to random (+2.75%), and even sits near the raw-pnl top-20 purged arm (−3.24%),
- loses to the baselines in 2 of 3 folds.
So the gate is mildly anti-selective: selecting on PnL / copy_score picks worse-than-random trades. This is the rigorous confirmation of the thesis already flagged in selector-vs-skill-gap and copy-trade-algorithm — the metric the selection path targets (cumulative PnL) does not persist, and here it actively hurts versus indiscriminate baselines.
The −43.37% was a leakage artifact
consensus-validation-leakage flagged the headline “−44%” gated-copy ROI as provisional (contaminated by lookahead / label leakage / survivorship). The clean re-derivation confirms it: the old leaky / unpurged gate measured −43.37%, but that catastrophic number was a LEAKAGE ARTIFACT, not a real out-of-sample loss. With leakage purged, the same gate is only mildly negative (−2.78%). The magnitude of the old number was an artifact of the contamination, not evidence of how badly the gate performs.
Why this harness is trustworthy — negative control ≈ 0
The decisive validity check is the negative control (the shuffled-label sentinel that was remedy step 4 in consensus-validation-leakage): run the whole pipeline on shuffled labels — any surviving “edge” would be leakage/artifact. It reads −0.04% ≈ 0, i.e. no spurious edge survives label-shuffling → the harness is leakage-clean. This is what lets us trust the −2.78% gate verdict and the baseline ordering as real signal rather than contamination.
The harness implements the consensus-validation-leakage remedy playbook:
- purged + embargoed walk-forward (45-day cutoff, 3 disjoint 15-day folds) — purge/embargo so trade→resolution overlap can’t leak across the cutoff (remedy step 3, CPCV);
- negative-control shuffled-label sentinel at ≈0 (remedy step 4);
- 200 bps slippage charged on fills.
Caveat — baselines are UPPER BOUNDS; trust the RELATIVE ordering
Baseline ROIs are upper bounds, not achievable returns
The baseline ROIs use last-trade price as a mid proxy, so they are UPPER BOUNDS — real fills would be worse than these numbers. Do not cite +3.68% (all-watched) or +2.75% (random) as achievable returns.
The robust finding is the RELATIVE ordering — gate < random < all-watched — not the absolute baseline magnitudes. The gate underperforming random/all-watched holds regardless of the mid-proxy optimism, because the same proxy is applied to every arm.
Implication — the Phase 2 bar
This sets a hard, quantified gate for best-consensus-algo-design: any new signal — the skill-gated selector (selector-vs-skill-gap) or the log-odds / independence aggregator (the skill-weighted logarithmic opinion pool) — must BEAT ~+2.75% random under this same purged harness to be worth shipping. This operationalizes the “required baseline arm” that best-consensus-algo-design already demanded: the bar is no longer “beats the old gate” (the old gate is anti-selective, a trivially low bar) but “beats indiscriminate random selection on a leakage-clean walk-forward.”
Performance fix — covering index for fetch_market_end_dates
A covering index turned the backtest’s slowest query from minutes into seconds:
idx_pm_pos_condition_enddate ON pm_positions (condition_id, end_date)fetch_market_end_dateswent from ~459s to an ~8s index-only scan (the(condition_id, end_date)covering index lets the planner satisfy the query without touching the heap).- Migration:
20260601120000_pm_positions_condition_enddate_idx.sql. - Applied
CONCURRENTLYto prod (no table lock) and is reversible. - Net effect: the full purged + control backtest now runs in ~2 minutes (previously dominated by the ~459s
fetch_market_end_datescall).
Covering-index pattern
When a backtest/analysis query filters/joins on
(condition_id, end_date)againstpm_positions, the covering indexidx_pm_pos_condition_enddategives an index-only scan (~8s vs ~459s). Build prod indexesCONCURRENTLYto avoid locking the live table.
Verified facts
| fact | value / location |
|---|---|
| harness | purged + embargoed walk-forward; 45d cutoff; 200 bps slippage; 3 disjoint 15-day folds |
| gate (recommended) pooled ROI | −2.78% |
| baseline: all watched | +3.68% |
| baseline: random | +2.75% |
| baseline: raw-pnl top-20 (purged) | −3.24% |
| negative-control (shuffled labels) | −0.04% (≈0 → leakage-clean) |
| old (leaky / unpurged) gate | −43.37% (leakage artifact) |
| folds where gate loses to baselines | 2 of 3 |
| baseline mid proxy | last-trade price → baselines are UPPER BOUNDS; trust relative ordering |
| Phase 2 ship bar | beat ~+2.75% random under this same purged harness |
| covering index | idx_pm_pos_condition_enddate ON pm_positions (condition_id, end_date) |
fetch_market_end_dates speedup | ~459s → ~8s index-only scan |
| migration | 20260601120000_pm_positions_condition_enddate_idx.sql (applied CONCURRENTLY to prod, reversible) |
| full purged + control backtest runtime | ~2 minutes |
Related
- consensus-validation-leakage — the doc that demanded this re-derivation and flagged the −44% gate as provisional; this note is its resolution. The negative-control ≈0 here is its remedy step 4 passing; the purged walk-forward is remedy step 3. The −43.37% is confirmed a leakage artifact.
- best-consensus-algo-design — the Phase 2+ algorithm whose ship bar this sets: it must beat ~+2.75% random under this purged harness. Operationalizes that doc’s “required baseline arm.”
- phase1-consensus-rebuild-implemented — Phase 1 consensus hygiene (drop redeemable/expired/dust, asset-keyed buckets); this purged verdict is the measurement side that Phase 1’s cleaner inputs feed into.
- selector-vs-skill-gap — predicted exactly this: the default selector targets cumulative PnL, which doesn’t persist; here PnL selection is shown to be anti-selective on a clean harness.
- copy-trade-algorithm — the original no-edge verdict (PnL doesn’t beat the luck baseline); this is its leakage-clean confirmation under purged walk-forward.
- polymarket-fetch — workspace layout,
pm_positionsschema, backtest harness, migrations. - TOPICS — project theme index.