Levandor

crypto-shortterm — Backtesting & Validation

5 min read

For Agents

The hardened validation funnel for crypto-shortterm (Rev 2). Leakage is prevented structurally by a causally-typed core (every input carries an as-of timestamp; future reads are impossible by construction), not by discipline. The bar extends beyond backtest: exec simulator → fault injection → dry-run → canary. Authoritative detail in the codebase spec/plan — see Canonical artifacts.

Working note for backtesting and validation methodology of crypto-shortterm.

No edge claim without a leakage-clean, cost-honest test

A strategy is “profitable” only if it survives an honest out-of-sample test and the live-realism gates below. Backtest alone cannot see clock skew, latency, rejects, partials, basis drift, or recovery — so it is necessary but not sufficient.

Causally-typed core (structural anti-leakage)

Live and backtest run the identical fairvalue/signal/risk code. The core consumes an immutable Snapshot carrying a hard as-of timestamp; constructing a snapshot from any input newer than as_of is an error. This makes lookahead impossible by construction rather than a thing reviewers must catch. Backtest just swaps the data source (recorded inputs replayed through the same core).

Backtest bar

  • Purged + embargoed walk-forward (disjoint folds; embargo around each cutoff).
  • Future-only, non-overlapping labels — enforced by the causally-typed core, not by hand.
  • Beat baselines net of costs: all-trades, random, price-follow.
  • Negative control ≈ 0 — shuffle the signal AND lag σ; if it still “profits,” the harness leaks.
  • Deflated Sharpe + significance (t ≈ 3) using a trial ledger that counts every parameter / variant swept (the multiple-comparisons honesty mechanism).
  • Pre-registered grid + a locked, never-tuned holdout.

Robustness

  • Per-regime (vol / trend) and rolling-period Sharpe with confidence intervals.
  • Transaction-cost sensitivity curve — must survive 1.5–2× modeled costs.
  • Power analysis on effective N, deflated by cross-window outcome correlation (overlapping 5-min windows are not independent).
  • Filled-edge vs signal-edge gap (adverse selection — are we getting filled mostly when wrong?).
  • Capacity curve specifically in the near-expiry window (where the strategy actually trades).

Cost model

Fills priced against recorded book depth (not mid), real fee(p) = C × 0.07 × p(1−p), plus simulated signing + network latency. No frictionless fills. (Fee model: Fees.)

Beyond backtest (live-realism gates)

The backtest can't see these — so we simulate them explicitly

  • Execution simulator: injected latency, rejects, partial fills, rate limits, 425/503 engine modes.
  • Fault injection: WS desync, clock skew, feed staleness, reconciliation mismatch.
  • Dry-run on the full live stack with stubbed / canary-sized orders against the real venue clock.
  • Minimal-size canary before any real ramp.

Promotion gate

backtest → exec-sim / fault-injection → dry-run → canary → small live (reduced Kelly) → ramp → maker mode (own gate). A journal compares realized vs expected at each gate (crypto-shortterm-trade-journal); degradation → demote & iterate.

Phase 0 analysis (runs first, no capital)

Before any trading backtest, the Phase 0 analysis binary replays recorded observe-only data to produce three measurements that gate whether Phase 1 is worth building:

  1. Calibration / reliability curve — bucket P_fair, compare to realized Chainlink Up-rate (expected calibration error). Decides whether the Gaussian P_fair is usable or the empirical overlay is required before sizing.
  2. Binance↔Chainlink basis distribution by τ-bucket and vol regime — is the basis small/stable enough near expiry to trade?
  3. Near-expiry touch-depth curve — is there investable size in the regime we’d actually use?

These feed the Phase 0 exit report (go/no-go for a Phase 1 taker plan, or pivot to maker). Plan tasks 18–19: /Users/levander/coding/pmv2/crypto_algo/docs/superpowers/plans/2026-06-08-phase0-observe-and-confirm.md.

analysis report-gen now runs (codebase commit 97f238d)

The analysis binary is built and runs against the crypto_shortterm data, writing docs/superpowers/phase0-report.md (crypto-shortterm-phase0-codebase-complete). The three measurements above are wired end-to-end — what remains is feeding it enough data.

First report is THIN — 2 windows, 0 outcomes, calibration pending

The first numbers are not decision-grade:

  • Calibration: pending — 0 closed outcomes yet (needs windows that close during a continuous run).
  • Basis-by-τ: near expiry ≈ -$40 (~6 bps) — real and sign-consistent, but a single point, not a distribution.
  • Near-expiry depth / book spread: spread ≈ 0.98 (bid ~0.01 / ask ~0.99) — books look empty/untradeable; this is the #1 result that could fail the go/no-go, and it echoes the ~$150–250 touch-depth finding. Could be a 2-window artifact.

The fix is data, not code: @deploy runs observe continuously across many windows, then re-run analysis. Until then no measurement here gates a decision.

Results

Pooled net ROI by arm vs baselines, per-fold breakdown, negative-control reading, cost-sensitivity curve, and the realized-vs-expected journal at each promotion gate. A strategy that does not clear the bar is shelved here with the verdict recorded.

  • Phase-0 first read (2026-06-08, THIN — 2 windows, 0 outcomes, do not over-conclude): basis near expiry ≈ -$40 (~6 bps); Polymarket book spread ≈ 0.98 (empty/untradeable — top risk); calibration pending (no closed outcomes). Details: crypto-shortterm-phase0-codebase-complete.
  • Full backtest results: TBD — pending the continuous-run data capture (@deploy) and the re-run analysis report.

Graph View

Backlinks