Phase-0 purged walk-forward test (K=6 sequential folds, embargo/purge at boundaries) against the full 219,688-row crypto_shortterm dataset. Verdict: mean PnL is positive across all folds but t-stat 1.46 falls well short of the multiple-testing bar (t≥3). EDGE IS FRAGILE / LIKELY OVERFIT — do NOT build live machinery yet.
Setup
- Binary:
crates/analysis/src/bin/walkforward.rs - Commit:
476cab1 - Report:
docs/superpowers/phase0-walkforward-report.md - Dataset: 219,688 rows, 899 windows (live
crypto_shorttermSupabase) - K: 6 sequential folds (purged walk-forward — each fold trains on all prior data, tests on the next block)
- Embargo/purge: 1 window removed at each fold boundary to prevent leakage across folds
- tau filter: τ≥60s (excludes the near-expiry leakage zone identified in crypto-shortterm-phase0-validation)
- Slippage: 1% (applied to every fill)
- Calibration: 20-bin isotonic calibration map (fit on train, applied to test)
- Per-window significance tracked
Out-of-Sample Results
| Metric | Value |
|---|---|
| Test windows | 750 |
| Active folds | 5 of 6 |
| Mean per-window PnL | +2.774130 |
| Std dev | 52.001 |
| t-stat | 1.4610 |
| Per-window Sharpe | 0.0533 |
| Folds positive | 5 / 5 |
| DSR vs N_trials=10 noise | ~0.0000 |
| Clears t≥3 bar | NO |
Verdict
EDGE FRAGILE / LIKELY OVERFIT — hold at Phase 0
t-stat 1.46 does not clear the multiple-testing bar of t≥3. Do NOT build live order-routing or position-taking machinery. This is the correct gate to enforce at this stage.
The prior single 70/30 split showing ~+0.8%/trade does not survive proper time-series cross-validation at the required significance level. The variance dominates the mean: stddev≈52 vs mean≈2.8 — the signal is too noisy relative to its size at the current data volume.
What is encouraging:
- 5/5 folds positive — the direction is consistent; this is not random noise flipping sign fold-to-fold.
- DSR probability vs N_trials=10 approaches zero, meaning the per-window Sharpe (0.0533) is unlikely to be a fluke of 10 independent searches, though this DSR is itself very small.
What is not sufficient:
- N_windows=750 is not enough to push a t-stat of 1.46 to ≥3 given this variance.
- The mean PnL is small relative to the window-level noise; the edge may be structurally small or the signal degrades in OOS conditions.
Implications for T19 Gate
T19 remains OPEN. The evidence trail now is:
- crypto-shortterm-phase0-edge-analysis — single 70/30 split showed +7.28% overall, +19.46% near expiry (leakage-contaminated, OPEN).
- crypto-shortterm-phase0-validation — ≥60s-gated re-run confirmed offline edge survives leakage debunk (+1.61%/trade at 0 slip, +0.81% at 1%), but single split, optimistic fills.
- This note — purged K=6 walk-forward with 1% slippage and τ≥60s filter: positive direction consistent (5/5 folds) but t-stat 1.46 < 3.0 — edge fragile / likely overfit.
Next Steps
Before any GO:
- More data — accruing more windows from the live observer will narrow the standard error. If the mean PnL holds at ~2.8, t≥3 requires reducing σ/√N below ~2.8/3 ≈ 0.93, which at σ≈52 means N≥3,100 windows.
- Stronger signal — feature engineering (OFI, OBI, vol regime filter) could raise the mean PnL or reduce per-window variance, shifting the t-stat.
- Depth / capacity read — needs book-size recording to model realistic fills beyond the 1% slippage proxy.
- Forward paper-trade — real-time fills with no capital before promotion to Phase 1.
For Agents
Status after this note: T19 OPEN. Walk-forward completed (K=6, 750 OOS windows, 1% slip, τ≥60s). t=1.46 < 3.0. 5/5 folds positive. Edge direction consistent but not statistically significant at the multiple-testing bar. Do not promote to Phase 1 or build execution machinery. Next hard gate: t≥3 on a larger dataset or a genuinely stronger signal.