Levandor

crypto-shortterm — Polymarket data-api Gotchas (activity harvest)

7 min read

Six reusable gotchas for harvesting a Polymarket wallet’s full trade history via data-api.polymarket.com/activity and for reading market tapes, plus how to read true economic PnL from the rows. Applies to any competitor / wallet study.

For Agents

Discovered building the wowitsamazing lifetime harvest (199,184 rows) and hardened by its 3-leg validation. This is a different Polymarket API surface than the Gamma/CLOB feeds in crypto-shortterm-data-sources — it is the read-only data-api activity/PnL/trades endpoints. Load-bearing facts: (1) /activity offset hard-caps at ~3000 → you must time-page; (2) usdcSize is the real USDC moved and includes the Polymarket taker fee 0.07·x·(1−x)·size (it is not price × size, and the gap is not a bid-ask spread) → use usdcSize for true PnL; (3) capped /trades?market= tapes are unusable for per-wallet PnL in BOTH directions; (4) the taker fee only exists after a cutover ~2026-06-13; (5) a rebate program makes the effective fee lower than the headline; (6) post-settlement tapes are frozen — exactly reproducible.

Gotcha 1 — /activity offset hard-caps at 3000

data-api.polymarket.com/activity?user=<w>&limit=500&offset=N returns

{"error":"max historical activity offset of 3000 exceeded"}

for any offset > 3000. You cannot paginate a full history by offset.

Workaround — time-page with start / end (unix seconds):

  • Rows come back newest-first; end is inclusive.
  • Page backward by setting end = min(timestamp) of the batch you just received; repeat until a batch comes back empty.
  • Parallelize by splitting the total time range into chunks and paging each chunk concurrently.
  • Needs a browser UA — curl -A "Mozilla/5.0" (the default curl UA is rejected).
  • Find the wallet’s first activity by binary-searching end=T: probe whether end=T returns any rows, bisect to the earliest T that does.

Recipe 

end = now
loop:
  GET /activity?user=<w>&limit=500&end=<end>    (header UA: Mozilla/5.0)
  if rows empty: break
  append rows
  end = min(row.timestamp) - 1

Split [first_ts, now] into K chunks and run K of these loops concurrently. Implemented in scripts/harvest_wallet_history.py (repo crypto_algo), which dedups by (tx, conditionId, asset, type, side, size, ts) and verified dollar-exact vs a fresh API fetch on the highest-volume days.

Gotcha 2 — usdcSize includes the Polymarket taker fee (not a bid-ask spread)

On /activity TRADE rows, usdcSize is the real USDC moved; the price field is the nominal (pre-fee) fill price. They differ systematically, and the gap is the Polymarket taker fee 0.07·x·(1−x)·size — not a bid-ask spread and not rounding:

  • For a buy, usdcSize > price × size (fee added on top); for a sell, usdcSize < price × size (fee taken out).
  • Independently verified as the fee, not a spread: per-fill ratio (usdcSize − price×size) / (0.07·x·(1−x)·size) ≈ 1.0, correlation 0.9915, model sum 87.5k (≈ 2.06% of notional) across ~190k fills.
  • For true economic PnL, use usdcSize — never price × size (which is pre-fee and flatters a taker).

This is the exact term in our net-edge gate

0.07·x·(1−x) is the same fee our validated live rule subtracts (walkforward.rs trade_decision_slipped: p_side − (ask + 0.01) − 0.07·eff·(1−eff) > 0; live as EMIT_MIN_EDGE / observe::net_buy_edge). The wowitsamazing harvest is a 190k-fill natural experiment showing a zero-edge taker loses exactly this fee — live validation that the gate’s fee subtraction is correct.

Polymarket's own PnL endpoints flatter a high-frequency taker

EndpointReportsBias
lb-api.polymarket.com/profit?...&interval=allleaderboard realized PnLcomputed off nominal price and/or live marks
user-pnl-api.polymarket.com/user-pnllive MTM PnLlive marks

Both appear to compute off nominal price / live marks, which excludes the taker fee actually paid. A high-churn taker therefore looks far better on Polymarket’s UI than his true USDC cash flow. Concrete gap (wowitsamazing): true USDC −7.6k vs nominal +$3.4k — all breakeven-to-negative, but the fee is the difference.

Redeem rows + the win-proxy caveat

  • REDEEM rows: usdcSize = size = $1/share; asset = '' and outcomeIndex = 999, but conditionId / slug are present → still attributable to a specific market.
  • Redeem-presence per conditionId is a valid per-market outcome proxy (validated 50/50 against CLOB tokens[].winner on the checked sample) — it tells you the market resolved such that a token the wallet held was redeemable.

Redeem-presence does NOT give a directional win RATE

Because a wallet can buy both sides, a hedged market always produces a redeem regardless of which way it resolved. So counting redeem-presence as “wins” overstates the directional win rate. In the wowitsamazing case he hedged 36.5% of BTC-5m markets, inflating the redeem-proxy rate to 79.3% when the true dominant-side rate (CLOB-truth labels) is 58.1%. Rule: use redeem-presence to build the per-market outcome ledger; use CLOB-truth dominant-side labels for any win rate.

Gotcha 3 — capped /trades?market= tapes are unusable for per-wallet PnL (BOTH directions)

The market-tape endpoint caps at 1000 prints per market, most-recent-first. On busy 5m markets (median 2,681 prints) that hides ~70% of a high-frequency wallet’s fills — and the errors go both ways:

  • Maker-invisibility: even a “taker” ladder bot executes a large share of fills as maker (resting ladder limits) — for wowitsamazing ~32% of fills executed as maker, so he is attributable as taker on only ~9% of his fills in capped tapes.
  • Bias direction is unpredictable: the same wallet read 242-0 in the wallet-aggregated 24h study, yet the capped tape’s visible fill-level slice showed him at 29% win rate vs 54% true. The cap censors wins and losses equally — the naive “the cap hid his losses” story was itself refuted (§C7); the exact 242-0 mechanism is being traced by exact reproduction of the frozen original tapes (§C8).

Rule

Never compute per-wallet skill/PnL from capped market tapes — the 24h study’s “sole Bonferroni survivor” per-wallet-z methodology is retracted entirely. Harvest the wallet’s own /activity lifetime instead (Gotcha 1). Archetype/band-level cell economics from tapes survive with explicit censoring caveats.

Gotcha 4 — the taker fee has a cutover date (~2026-06-13)

Fills before ~Jun 13, 2026 carry no taker fee (usdcSize ≈ price × size). A zero-fee subset in a harvest is the pre-fee era, not a data anomaly and not evidence against the fee identity — split any fee analysis by era. (wowitsamazing pre-cutover: −4.1k nominal − $87.5k fee.)

Gotcha 5 — rebates make the effective fee lower than the headline

A maker/taker rebate program claws back part of the fee: for wowitsamazing **+378 + taker $14,028) ≈ 16% of fees paid → 1.72% effective vs the raw 2.06% of notional. Rebates arrive as their own reward rows.

Follow-up for our own book

Check whether OUR wallet accrues taker rebates. If yes, our realized costs are better than the model in the net-edge gate assumes — free EV, zero code.

Gotcha 6 — post-settlement tapes are frozen (exactly reproducible)

Re-querying a settled market’s tape returns row-exact identical data (validated 10/10 live re-queries). Consequences: (1) freeze tapes at study time and old studies become exactly reproducible later — this is what enables the §C8 mechanism trace of the 242-0 artifact; (2) a reproduction mismatch on a frozen tape means your pipeline changed, not the data.

Shared-file clobber incident (2026-07-03)

dissect_discriminator.py briefly overwrote data/0xd02b_activity_full.csv with an 8h slice; two validators read the stub during the ~15-min window (restored from raw JSON; script repointed to 0xd02b_activity_recent8h.csv). Check file mtime when a claim contradicts a recent measurement, and give derived slices their own filenames.

Graph View

Backlinks