Phase 1 Consensus Rebuild — Implemented & Verified (Uncommitted)

Phase 1 of the consensus rebuild is implemented and verified green (uncommitted working tree, 2026-06-01). Two workstreams landed: WS1 adds consensus eligibility hygiene (drop redeemable / past-end_date / dust positions, and key buckets by asset so opposite-asset positions can’t inflate a long outcome’s wallet count); WS2 wires the daily heavy-sync (short-term-markets + short-term-accuracy) into the scheduler behind a DB-backed once-per-24h gate. This is the first concrete code response to the leakage/selection findings in consensus-validation-leakage and selector-vs-skill-gap — it cleans what counts as a consensus voice, ahead of the full point-in-time signal rebuild that best-consensus-algo-design targets.

For Agents

Verified green this session (uncommitted): cargo fmt clean, cargo clippy no issues, WS1 = 13 tests pass, WS2 = 4 tests pass.

• WS1 (crates/polymarket-data/src/consensus.rs): new compute_consensus_groups_with(members, min_wallets, now, dust_floor_usd); the public compute_consensus_groups delegates with Utc::now() + DEFAULT_DUST_FLOOR_USD = 5.0. New eligibility filter drops redeemable, past-end_date, and dust (current_value < 5.0) positions. Bucket key changed (condition_id, outcome_index) → (condition_id, outcome_index, asset).
• WS2 (crates/polymarket-fetch/src/scheduled_run.rs + main.rs + repo.rs): scheduler now runs short-term-markets-sync + short-term-accuracy in dependency order, gated to ≤ once per 24h via a DB-backed gate (heavy_sync_is_due, reads max(computed_at) from pm_short_term_trader_accuracy). The 30-min consensus_refresh path is unchanged.
• Two follow-ups: (1) DRY — parse_end_date is now duplicated in consensus.rs and repo.rs across crates → hoist into polymarket-data. (2) Behavior change to watch — asset-in-key + $5 dust floor will reduce consensus group counts (correctly dropping dead/dust/synthetic-short).

WS1 — Consensus eligibility hygiene

File: crates/polymarket-data/src/consensus.rs.

New testable seam: compute_consensus_groups_with

The existing public entry point compute_consensus_groups now delegates to a new, fully-parameterized inner function:

compute_consensus_groups_with(members, min_wallets, now, dust_floor_usd)

The public wrapper supplies the production defaults — Utc::now() for now and DEFAULT_DUST_FLOOR_USD = 5.0 for dust_floor_usd. Injecting now and the dust floor as parameters is what makes the eligibility logic deterministically testable (fixed clock, fixed floor) — the 13 WS1 tests drive this seam.

Eligibility filter — three exclusions

A position is excluded from consensus aggregation if any of:

1. Redeemable — the position is already redeemable (the market has resolved in its favour and it’s claimable); it is settled history, not a live opinion.
2. Past end_date — the market’s end_date is at or before now. The date is parsed leniently: first try the bare-date form %Y-%m-%d, then fall back to RFC3339. An unparseable end_date is kept (fail-open) and a tracing::warn! is emitted so the bad value surfaces in logs rather than silently dropping the position.
3. Dust — current_value < dust_floor_usd (production floor $5.0). Tiny residual positions carry no real conviction and shouldn’t count as a wallet voting for an outcome.

end_date parse order is deliberate

%Y-%m-%d is tried before RFC3339, and an unparseable value is kept + warned, not dropped. Fail-open (keep on parse failure) avoids silently shrinking consensus on a formatting quirk; the tracing::warn! is the safety valve that makes the bad data visible.

Bucket key now includes asset

The aggregation bucket key changed:

(condition_id, outcome_index)  →  (condition_id, outcome_index, asset)

Why the asset dimension matters — negative-risk / opposite-asset inflation

Without asset in the key, a wallet holding the opposite asset (e.g. the negative-risk / short side, or a different token on the same condition+outcome index) could be bucketed together with longs and inflate that outcome’s wallet_count. Adding asset ensures only wallets actually long the same asset on the same outcome aggregate together — the count now reflects genuine same-side agreement, not a mix of opposing exposures.

WS2 — Daily heavy-sync, DB-backed once-per-24h gate

Files: crates/polymarket-fetch/src/scheduled_run.rs, main.rs, repo.rs.

Dependency-ordered heavy pipeline

The scheduler now runs the heavy short-term pipeline in strict dependency order:

trades → resolutions → short-term-markets → metrics → short-term-accuracy

Each stage feeds the next: trades and resolutions must be current before short-term-markets can be derived; metrics before short-term-accuracy can score wallets. This is the same short-term-accuracy skill layer (pm_short_term_trader_accuracy) that selector-vs-skill-gap flags as the rigorous-but-unwired measurement layer — WS2 makes it run automatically on a daily cadence instead of only on manual CLI invocation.

The gate is DB-backed, not in-memory — and why

The heavy pipeline is gated to run at most once per 24h by heavy_sync_is_due, which checks max(computed_at) in pm_short_term_trader_accuracy and returns due/not-due.

DB-backed gate is a deliberate choice, not an oversight

The scheduler is a fresh process on each ~30-minute tick (systemd timer OnCalendar=*:0/30, see polymarket-fetch-deploy). An in-memory “last run” timer would reset every tick and never persist, so the heavy sync would either run every 30 min or never gate correctly. Reading max(computed_at) from the table the pipeline itself writes makes the gate survive process restarts for free — the last successful heavy run timestamps itself, and the next tick reads it back.

30-min path untouched

The existing 30-minute consensus_refresh path is unchanged by WS2 — the heavy sync is layered alongside it, gated independently. Each ~30-min tick still refreshes consensus; only the heavy short-term pipeline is throttled to daily.

Follow-ups (tracked, not yet done)

DRY — hoist parse_end_date into polymarket-data

parse_end_date (the %Y-%m-%d-then-RFC3339 lenient parser) is now duplicated across crates: in crates/polymarket-data/src/consensus.rs and in crates/polymarket-fetch/src/repo.rs. It should be hoisted into polymarket-data and reused from both. Pure duplication — extract before it drifts.

Behavior change to watch — consensus group counts will drop

The combination of asset-in-key + the **$5dustfloor\ast \ast will\ast \ast reduce\ast \ast thenumberofconsensusgroupsproduced.Thisis\ast \ast correct,notaregression\ast \ast :itdropsdeadmarkets(past‘en{d}_{d}ate‘),dustpositions(‘<$5`), and synthetic-short / opposite-asset positions that previously inflated long-outcome counts. Expect lower group counts after this lands; don’t mistake the drop for a bug.

Status & verification

Verified green (uncommitted, 2026-06-01)

• cargo fmt — clean
• cargo clippy — no issues
• WS1 (consensus.rs) — 13 tests pass
• WS2 (scheduler/gate) — 4 tests pass

Working-tree only — not committed, not deployed. Deploy note: the scheduled-run unit is the systemd-timer process that will pick up WS2 (polymarket-fetch-deploy).

Verified file:line references

fact	location
new parameterized seam compute_consensus_groups_with(members, min_wallets, now, dust_floor_usd)	crates/polymarket-data/src/consensus.rs
public compute_consensus_groups delegates with Utc::now() + DEFAULT_DUST_FLOOR_USD = 5.0	crates/polymarket-data/src/consensus.rs
eligibility filter (redeemable / past-end_date / dust < 5.0)	crates/polymarket-data/src/consensus.rs
end_date parse order %Y-%m-%d then RFC3339; unparseable kept + tracing::warn!	crates/polymarket-data/src/consensus.rs
bucket key (condition_id, outcome_index) → (condition_id, outcome_index, asset)	crates/polymarket-data/src/consensus.rs
dependency-ordered heavy pipeline (trades→resolutions→short-term-markets→metrics→short-term-accuracy)	crates/polymarket-fetch/src/scheduled_run.rs (+ main.rs)
DB-backed once-per-24h gate heavy_sync_is_due (max(computed_at) in pm_short_term_trader_accuracy)	crates/polymarket-fetch/src/repo.rs (+ scheduled_run.rs)
parse_end_date duplicated (DRY follow-up)	crates/polymarket-data/src/consensus.rs and crates/polymarket-fetch/src/repo.rs

Related

• consensus-validation-leakage — leak #1 was consensus.rs aggregating LIVE positions; this WS1 hygiene (drop redeemable/expired/dust, asset-keyed buckets) is the first cleanup of what counts as a voice, ahead of the full point-in-time rebuild that work demands
• selector-vs-skill-gap — WS2 auto-runs the short-term-accuracy skill layer (pm_short_term_trader_accuracy) daily; that table is the one an edge-based selector should draw from
• best-consensus-algo-design — the Phase 2+ target this is Phase 1 toward; the fresh point-in-time forecast layer there supersedes live-snapshot aggregation entirely
• polymarket-fetch — workspace layout, compute_consensus_groups, scheduler, pm_short_term_trader_accuracy schema
• polymarket-fetch-deploy — the scheduled-run systemd-timer process (OnCalendar=*:0/30) that runs the WS2 gate; fresh process per tick is why the gate is DB-backed
• TOPICS — project theme index