Levandor

solv1-alpha GMGN-Ingester Increment — Browser-Harvest → File-Ingest

6 min read

The chosen next increment for solv1-alpha after the data-source pivot. Because Cloudflare blocks non-browser clients, ingestion becomes a browser-harvest-to-JSON step (Claude drives a same-origin fetch() in claude-in-chrome) feeding a Rust file-ingester that lands rows in solv1_alpha_trades; the analysis stack A8–A16 is reused unchanged. Full scope chosen 2026-07-04 — and now BUILT (G2 858b5b1, G3 ac2b6c1, G4 d4e6998; 56 tests; migration 0003_solv1_alpha_klines). The first run on this path produced the project’s first verdict: NO-GO on naive copy-trading.

Status: BUILT (2026-07-04)

All tasks landed. Two things changed vs the plan below: (1) decay is now priced from GMGN 1-second klines, not per-fill streams — see Decay Pivot — per-fill streams → 1s klines; (2) the harvest bridge is a blob download to ~/Downloads, not a localhost POST — see Harvest Bridge (as built). The two-phase structure and the reuse of A8–A16 held exactly as designed.

For Agents

This increment replaces RPC-indexing tasks A4–A7 with a browser-harvest + file-ingest path — see solv1-alpha-data-source-pivot-2026-07-04 for why (non-archival Chainstack node + bot-polluted leaderboard). It is deliberately semi-manual and two-phase: an agent harvests one batch of JSON, a Rust stage consumes it, then — once the cohort is known — the agent harvests a second batch (the cohort tokens’ full fill streams) for the decay/mirror stage.

Scope = Full (chosen over a thinner slice): all five tasks G1–G5 below. The bot/dead filter that Root-Cause-2 of the pivot demands lives in G1 (activity-rate + GMGN smartmoney flag), so the leaderboard pollution is handled at harvest time, before scoring.

Why browser-harvest (the constraint)

Rust cannot call GMGN directly: headless curl/reqwest get a Cloudflare 403, and only a real browser context with a live cf_clearance cookie passes. So the network fetch runs inside claude-in-chrome (same-origin fetch() against /api/v1/wallet_activity/sol?wallet=<addr>&cursor=<data.next>, ~20 events/page, cursor-paginated), writes JSON to disk, and the Rust side reads files, never sockets. Full endpoint/schema detail: solv1-alpha-data-source-pivot-2026-07-04.

Two-Phase Run

The run is split because the second harvest can’t be enumerated until the first analysis picks the cohort — you only know which tokens’ fill streams to pull after scoring/filtering/cohort-selection has named the wallets and their tokens.

graph TD
    subgraph P1["Phase 1 — wallets"]
        H1["harvest wallet_activity<br/>(browser · G1)"] --> I1["ingest → solv1_alpha_trades<br/>(G3)"]
        I1 --> SC["score → filters → cohort<br/>(A8–A12, reused)"]
    end
    SC -->|"cohort wallets + their tokens"| P2
    subgraph P2["Phase 2 — cohort-token fills"]
        H2["harvest full fill streams<br/>for cohort tokens<br/>(browser · G4)"] --> I2["ingest fills (G4)"]
        I2 --> MV["mirror (real decay) → benchmarks → verdict<br/>(A13–A16, reused)"]
    end
    style SC fill:#2d2d2d,stroke:#e9c46a,color:#fff
    style MV fill:#2d2d2d,stroke:#e9c46a,color:#fff
    style H1 fill:#264653,stroke:#2a9d8f,color:#fff
    style H2 fill:#264653,stroke:#2a9d8f,color:#fff

Phase 2’s per-token fill streams are what let the paper-mirror model real entry/exit decay from the token’s actual subsequent on-chain fills (rather than a fixed haircut) — the same decay contract the built harness already implements.

Tasks (G1–G5)

TaskDeliverableCommitNotes
G1Harvester — drive same-origin fetch() over wallet_activity, cursor-paginate, write JSON per wallet—Includes the bot/dead pre-filter (activity-rate + GMGN smartmoney flag) that Root-Cause-2 of the pivot requires — pollution is dropped at harvest, before scoring. In the first run it auto-rejected 84 of 121 harvested wallets as bots
G2JSON→Trade mapper — map the GMGN event schema to the harness Trade type858b5b1synthetic slot ← timestamp; buy/sell sign; quote_amount as SOL notional; priority_fee + tip_fee; dex venue
G3Wallet ingest + Phase A — file-ingester loads harvested wallet-activity JSON into solv1_alpha_trades, runs score/filters/cohortac2b6c1Phase-1 sink; feeds score/filters/cohort
G4Kline-decay + Phase B — harvest 1-second klines per cohort token, ingest into solv1_alpha_klines, run mirror/bench/verdictd4e6998Pivoted from per-fill streams to 1s klines (per-fill harvest of 391 tokens was infeasible; klines give ~100% coverage cheaply). Adds migration 0003_solv1_alpha_klines. See
G5run_all rewire — wire the two-phase flow into orchestration so score→verdict runs on ingested data(folded into G3/G4)Sequences Phase 1 → cohort → Phase 2 → verdict

Build totals: commits 858b5b1 · ac2b6c1 · d4e6998, 56 tests green, migration 0003_solv1_alpha_klines added to the solv1_alpha_* schema.

Decay Pivot — per-fill streams → 1s klines

The plan’s Phase 2 was to harvest each cohort token’s full subsequent-fill stream so the paper-mirror could model follower entry/exit decay from actual on-chain fills. That turned out to be infeasible at scale: the first run touched 391 tokens, and a full per-fill harvest of all of them blows past GMGN’s session rate limit (see below).

The fix: price decay from GMGN 1-second klines instead of raw fills.

Kline decay endpoint 

GET /api/v1/token_kline/sol/<token>?resolution=1s

One request per token returns a dense 1-second OHLC series — enough temporal resolution to model a follower’s few-hundred-ms-late fill against the real price path, at ~100% token coverage for a single cheap request per token instead of paginating an unbounded fill stream. Landed in solv1_alpha_klines (migration 0003). In the first run this priced 231 positions with real decay.

This is a strictly cheaper substitute for the same decay contract the built harness expects — the mirror still models real price movement between the leader’s fill and the follower’s, it just reads it off a kline bar rather than off the next raw fills.

Harvest Bridge (as built)

Getting harvested JSON from the browser to disk was the fiddly part. The as-built bridge:

  • Browser same-origin fetch() → blob → download to ~/Downloads. The page fetches GMGN in-context (carrying cf_clearance), wraps the response in a Blob, and triggers a browser download. The Rust ingester then reads the files out of ~/Downloads.
  • A localhost sink does not work — CSP blocks it. Posting the harvested JSON to a local http://localhost collector is Content-Security-Policy-blocked from the GMGN origin, so the blob-download path is the only egress that clears both Cloudflare (inbound) and CSP (outbound).
  • One-time permission: the browser download path needs the site’s “automatic downloads” permission granted once, after which the loop runs unattended.
  • Fire-and-forget async loop: a page-side async loop harvests unattended (no per-request babysitting) — kick it off, let it drain.
  • Rate ceiling: GMGN throttles at ~1.3k fetches/session. This is the hard cap that made the 391-token full-fill harvest infeasible and forced the 1-request-per-token kline pivot above.

The rate limit is the design constraint

GMGN rate-limits at ~1,300 fetches per session. Any Phase-2 design that scales per fill (unbounded pages per token × hundreds of tokens) will exhaust the session; the 1-request-per-token kline design fits inside the budget. Plan future harvests around this ceiling, not around ideal data richness.

Reused vs Replaced

  • Replaced (A4–A7): rpc.rs, index.rs, swap.rs, universe indexing → G1–G4 (harvest + file-ingest).
  • Reused unchanged (A8–A16): score → filters → cohort → decay → mirror → benchmarks → verdict. These read solv1_alpha_trades and don’t care how it was filled — which is exactly why the pivot is cheap.

Semi-manual by design, not by omission

There is a human/agent-in-the-loop harvest between the Rust stages (and again between Phase 1 and Phase 2). This is intentional — it is the price of clearing Cloudflare with a real browser session. It is not a candidate for “just call the API from Rust”; that path is the 403 the pivot ruled out.

Graph View

Backlinks