crypto-shortterm — Phase 0 Observe Pipeline Live (Milestone)

For Agents

Milestone (2026-06-08, commit f50b084): the Phase-0 observe-only pipeline is BUILT and VERIFIED LIVE end-to-end. The observe binary ran all four feeds and wrote to the isolated crypto_shortterm Supabase schema for ~5 minutes. This unblocks the Phase-0 run — the prior “blocked on access” status (Supabase URL, Chainlink Data Streams, run env) is resolved. Settlement reference is now an on-chain Chainlink read on Polygon via Alchemy RPC — NOT the paid Data Streams product. No capital committed.

The first end-to-end live run of the crypto-shortterm Phase-0 observe pipeline. Records what was built, what the run proved, and the three findings that change downstream work.

What ran

The observe binary started all four feeds and the async recorder, then ran live against production venues for ~5 minutes, writing to the isolated crypto_shortterm Supabase schema. Commit f50b084.

Live architecture (all four feeds verified writing)

• Binance @trade WS → predictor price (Ŝ, basis, σ).
• Chainlink BTC/USD read on-chain on Polygon via Alchemy RPC (~1s poll) → settlement reference (K at window open, settle at close, basis = Chainlink − Binance).
• Polymarket 5-min market discovered via Gamma (btc-updown-5m-<300s-aligned-ts>) + CLOB /book polled ~1s → book top.
• Clock offset via Binance server-time (lenient 5s bound for observe-only; strict 0.25s is reserved for Phase-1 trading).
• recorder writes async to crypto_shortterm.

Run results (~5 min)

Table	Rows	Note
binance_ticks	3,805	Raw trade ticks (throttled — see Finding 2).
cl_reports	28	Chainlink on-chain reads.
pm_book_snaps	728	Polymarket CLOB book tops.
observations	1,340	k = 63,596 set at a witnessed window open; p_fair evolved correctly toward expiry.
outcomes	0	Needs a full window to close during a sustained run (the run did not span a clean close).

What the observations row proves

k was set to 63,596 at a window whose open the process witnessed, and p_fair evolved correctly toward expiry — i.e. the fair-value model is wired end-to-end on live data (Chainlink K, Binance-derived Ŝ, σ, τ → P_fair). This is the first live confirmation that the P_fair pipeline computes sensibly on real feeds.

Architecture change — Chainlink via on-chain RPC, not Data Streams

The Data Streams blocker is gone

Earlier notes listed “Chainlink BTC/USD Data Streams access (no public free tier)” as a hard Phase-0 blocker. The live build instead reads the Chainlink BTC/USD aggregator on-chain on Polygon via an Alchemy RPC (~1s poll). This removes the paid-Data-Streams dependency entirely for observe, and is the actual settlement-reference path now in code. The remaining CONFIRM is still which Chainlink report timestamp Polymarket samples for open/close (gate 1 in crypto-shortterm-open-questions) — the access sub-problem is solved.

Key findings

1. CONFIRMED — Polymarket DOES have 5-minute BTC Up/Down markets

Series btc-up-or-down-5m, slug btc-updown-5m-<ts>, 300s windows. An earlier build attempt wrongly pivoted to 15-minute because a timestamp-alignment bug floored to 900s; corrected to 300s. This closes the lingering doubt about whether the 5-min instrument exists — it does, and the live discovery via Gamma works.

Timestamp-alignment bug (fixed)

The window-open timestamp must be floored to 300s, not 900s. The 900s floor silently produced 15-min slugs and led a prior build down the wrong path. Any future alignment math (and the K-at-open logic) depends on the 300s floor being correct.

2. Recorder throughput saturates on raw Binance ticks — Phase-1 blocker

Important for Phase 1 — async per-row inserts can't keep up

Raw Binance ticks (~50/s) saturate the recorder’s per-row async inserts even at channel cap 20k, so rows spill instead of insert. Mitigation for Phase 0: throttle raw + observation recording to 1/s (fine for the 1s-cadence calibration Phase 0 needs). Phase 1 — and any finer-grained basis/σ work — needs batched / COPY inserts instead of per-row. Overflow may exist at /tmp/crypto_shortterm_spill.jsonl.

This is consistent with the design split (the async recorder is telemetry only and may drop/spill — see Architecture); the milestone makes the throughput limit concrete and dates the batched-insert work to before finer basis/σ or Phase 1.

3. Window K-correctness requires witnessing the open

K is only set for windows whose OPEN the process witnessed. A mid-window start skips that window (no K → no usable P_fair/outcome for it). Consequence: meaningful data only accrues when the process runs across clean window boundaries — so the observe loop must run continuously to harvest many complete windows, not in short ad-hoc bursts.

Status impact

This moves crypto-shortterm from “Phase 0 — logic built, blocked on access” to “Phase 0 — observe pipeline live & verified; accumulating data.” The remaining Phase-0 work is now operational + analysis, not access provisioning:

1. Run observe continuously (local or deployed) so observations and outcomes accumulate across many complete windows.
2. Build/run the analysis report — reliability curve (P_fair vs realized), basis-by-τ, depth-by-τ.
3. Phase-0 exit / gate decision against the CONFIRM gates (crypto-shortterm-open-questions).
4. Then Phase 1.

A funded account is still needed only for the deferrable fee-at-match check (gate 3); it does not block continued observing.

Related

• crypto-shortterm — project index (status reflects this milestone).
• crypto-shortterm-data-sources — the four feeds, the Supabase schema, the recorder-throughput note.
• crypto-shortterm-strategy-design — P_fair model now confirmed live; clock-bound and recorder-as-telemetry context.
• crypto-shortterm-open-questions — CONFIRM gates; Data-Streams-access blocker resolved.
• crypto-shortterm-backtesting — the analysis report this run feeds (reliability curve, basis-by-τ, depth-by-τ).