Levandor

solv1-rig Increment 1 — Build Complete

7 min read

Increment 1 of solv1-rig — a Rust dual-feed Solana latency-measurement rig — is built and green. Its whole job is to produce self-measured shred-vs-Yellowstone-gRPC arrival-delta data, the kill-criterion the copy-trading feasibility left unproven. 15 tasks, subagent-driven on Opus, every task individually reviewed plus a final whole-branch review returning “ship it” (0 critical).

For Agents

This rig is measurement only — no transaction is ever signed or sent. The trustworthiness of the one number it exists to produce (delta_ms) rests on five load-bearing decisions (see below); do not “simplify” any of them without re-reading why. Source of truth for full detail is the spec + plan (local repo paths in Related) — this note summarizes and links, it does not duplicate them.

Key facts: histogram is the percentile authority (raw rows are 1:100 sampled debug data); the Yellowstone leg is venue-classification authority (shred side is a best-effort hint, blind to ALT-routed swaps); delta_ms = geyser_at − shred_at, positive = shreds arrived earlier. Increment 1 uses a single-consumer mpsc pipeline (not the spec’s broadcast bus — deferred to increment 2). Storage is the Supabase mgmt project (mkofmdtdldxgmmolxxhc), 8 solv1_-prefixed tables, via session-mode pooler :5432.

What Shipped

One crate, one binary (solv1-rig, tokio async), three subcommands:

  • run — the dual-feed collector (the deliverable).
  • report — SQL percentile readout per venue/hour from the histogram authority, with an n-floor that suppresses under-sampled cells.
  • capture — records ~5 min of live feed frames to a fixture file for the replay tests / shakeout.

15 tasks (scaffold+pinned deps → config → venue classifier → histogram → matcher+stats → migrations → sink → tips → FeedSource trait+ReplayFeed → RPC Fast adapter → Yellowstone adapter → orchestration → e2e → report → Docker/CI). Subagent-driven development on Opus; each task individually reviewed, then a final whole-branch review = “ship it”, 0 critical findings.

Architecture

Feed-agnostic by hard requirement (Jito ToS is revocable — swapping a feed must be a config change, never a code change). Full design is spec §3; the increment-1 realization:

Increment 1 data path (single-consumer mpsc) 

graph LR
    RF["RpcFastShreds<br/><i>stamp Instant pre-decode</i>"] --> BUS{{"mpsc TxSeen<br/>(1 consumer)"}}
    YS["Yellowstone<br/><b>venue authority</b>"] --> BUS
    BUS --> M["matcher<br/>cross-feed sig pairing"]
    M --> H["minutely histograms<br/><b>percentile authority</b>"]
    M --> RAW["1:100 sampled<br/>raw rows"]
    H --> SINK["batched sqlx sink<br/>500 rows / 2s, retry×3"]
    RAW --> SINK
    TIPS["Jito tip-floor poller<br/><i>minutely</i>"] --> SINK
    SINK --> PG[("Supabase PG<br/>mgmt · solv1_*")]
    style RF fill:#264653,stroke:#2a9d8f,color:#fff
    style YS fill:#3d2020,stroke:#c1666b,color:#fff
    style H fill:#2d2d2d,stroke:#e9c46a,color:#fff
    style PG fill:#2d2d2d,stroke:#888,color:#fff

Cohort-watch + paper-mirror pricer are the increment-2 consumers that don’t exist yet — see Increment-2 Backlog.

Components (all behind the feed::FeedSource trait seam where feed-facing):

  • FeedSource trait — subscribe() yields TxSeen { feed, sig, slot, arrived: Instant, venue, fee_payer }. arrived is stamped at gRPC message receipt before any decode. Increment-1 impls: RpcFastShreds (SubscribeEntries → bincode Vec<Entry> → VersionedTransactions) and Yellowstone (SubscribeRequestFilterTransactions, commitment Processed, echoes server pings). JitoDirect and any replacement vendor slot in later by config only.
  • matcher — bounded map keyed by signature; all shred events enter (not venue-pre-filtered — the geyser leg is server-side venue-filtered, so a completed match implies a venue tx and inherits the geyser leg’s venue). 30s TTL; expiries split unmatched_hinted (alarming) vs unmatched_other (expected bulk); 1M cap with counted eviction; per-feed flush on reconnect so stale first-sightings can’t fabricate deltas.
  • hist / sink / tips / report — minutely MinuteHist (1ms buckets 0–2000ms + under/overflow, percentile-by-bucket), batched sqlx writer (bounded mpsc → 500-row/2s INSERT batches, 3 retries then drop-with-counter), minutely Jito tip_floor poller, and the histogram-percentile readout.

Load-Bearing Design Decisions

These are what make the measured delta_ms trustworthy. Ordered by how badly a “simplification” would corrupt the data:

  1. Pre-decode Instant timestamps on BOTH feeds, one monotonic clock, one box. arrived is stamped at message receipt before any protobuf/bincode decode, on both adapters, on a single machine — so the delta is pure monotonic arithmetic and NTP drift can’t touch it. Adapter asymmetry (Yellowstone’s client lib decodes protobuf before yielding) is ≤1ms, negligible against the 50ms kill threshold. This is the integrity of the entire measurement.
  2. The Yellowstone leg is venue-classification authority — it classifies over static plus ALT-resolved keys (loaded_writable / loaded_readonly), closing the router-wrapped / ALT blind spot that the shred side structurally has. The shred side computes only a best-effort venue_hint from static keys. Full trap write-up: solana-v0-alt-classification-trap.
  3. The histogram is the percentile authority; raw rows are 1:100 sampled. Percentiles (p50/p90/p99) come from constant-size solv1_delta_hist buckets, never from raw rows. This is what lets the rig survive the corrected 10⁷–10⁸ matched tx/week volume (an earlier estimate was 10–100× low) — persisting every match would be infeasible on a standard Supabase plan.
  4. delta_ms = geyser_at − shred_at; positive = shreds arrived earlier. The sign convention is the whole point (does the shred feed actually lead commodity gRPC?) — it is asserted in the matcher tests.
  5. No std::sync::MutexGuard held across any await. Learned mid-build while writing the adapters: a non-Send guard alive across an await point makes the future non-Send and won’t tokio::spawn. Adapter state is dropped/scoped before await, or an async-aware lock is used.

Two more decisions worth knowing

  • Increment 1 is an mpsc pipeline, not the spec’s broadcast bus. With a single consumer (the matcher) mpsc is correct and simpler; broadcast only earns its keep once cohort/mirror become a second consumer (increment 2).
  • DB connection is Supavisor session mode :5432 only — config hard-fails on a :6543 URL (transaction mode breaks sqlx prepared statements). See supabase-from-gcp-connection-gotchas.

Storage

  • Supabase mgmt project (ref mkofmdtdldxgmmolxxhc), all tables prefixed solv1_ — 8 tables: solv1_runs, solv1_delta_hist, solv1_tx_deltas, solv1_feed_stats, solv1_wallets, solv1_fills, solv1_mirror_decay, solv1_tip_snapshots (the last three seed increment 2).
  • RLS on / FORCE off / anon-denied. RLS enabled with zero policies (PostgREST roles default-deny); the rig connects as postgres and writes because it owns the tables — never FORCE ROW LEVEL SECURITY (that would subject the owner to the empty policy set and deny all writes). Defense-in-depth REVOKE ALL … FROM anon, authenticated.
  • Applied live via the Supabase MCP and mirrored as migrations/0001_solv1_init.sql in-repo. MCP is a dev/ops tool, never on the write path — the rig writes via plain DATABASE_URL.
  • Full pooler-port / RLS-via-ownership rationale: supabase-from-gcp-connection-gotchas.

Verification

  • 21 tests green — matcher (dup sig, TTL hint-split sweep, reconnect flush, cap eviction, delta sign), venue classifier, histogram buckets/percentiles/merge, config :6543 guard, sink batching, plus adapter and tips unit tests.
  • Pipeline proven END-TO-END against a live throwaway Docker Postgres: ReplayFeed → matcher → histogram → sink → read-back produced pumpfun 121ms / pumpswap 155ms histogram rows; run_with_feeds returned cleanly (proving the T12 shutdown-deadlock fix — tips poller is now aborted on shutdown so the run doesn’t hang); report rendered the percentile table.
  • Docker image builds (release binary compiled in-container) and runs fail-fast without secrets (no silent no-op collection when DATABASE_URL / tokens are absent).

Status & Blockers

  • On git main, local, unpushed (no remote configured). 15 feature commits + fixups.
  • Cannot run for real yet — two external dependencies gate live shakeout:
    1. RPC Fast credentials (owner: levander) — real ShredStream + Yellowstone endpoints into env.
    2. GCP europe-west3 e2-standard-2 VM (deploy via babylon) — e2-standard-2 specifically, because a shared-core burstable instance would add CPU-throttle jitter to the arrival-Instant stamps, biasing the exact quantity being measured.
  • Everything except live capture works today (the e2e proof ran against Docker Postgres with a ReplayFeed).

Increment-2 Backlog

Carried forward deliberately (deferred, not missed):

  • Cohort watch + paper-mirror decay — the second kill criterion (entry-decay vs modeled copy PnL); needs seeded solv1_wallets, pool-state tracking, and swap-instruction parsers.
  • Shutdown-tail on conflict do update — the final partial (<60s) minute of histogram/stats is currently dropped on shutdown; upsert-merge it.
  • mpsc → broadcast bus — flip when cohort/mirror become the second consumer of TxSeen.
  • Second-shred-feed (Jito direct) guards — FeedId::is_shred / feed_pair currently assume a single shred feed; add guards before wiring JitoDirect.
  • report timestamp column formatting — cosmetic.

Graph View

Backlinks