Levandor

Entry-Path Config Cache — Design Corrections That Mattered

5 min read

PM runs in Zurich; Supabase management sits in Dublin — ~183ms per sequential round-trip — and config reads were the single largest controllable stage on the live entry path. This note records the config-cache work and, more importantly, the eight design corrections an adversarial review found BEFORE building — several of which would have shipped a stale cache that silently placed unauthorized live orders. Branch: feat/entry-config-cache @ ec0b666.

For Agents

The headline reusable idea is at Key architectural idea — move the authority check to a round-trip you already pay: when you cache authority, move the authority check onto a round-trip you already pay (here, the fast_reserve INSERT became a CTE conditional on the DB’s own mode/breaker_tripped). This collapsed the whole design — no generation counter, no breaker snapshot-patch, no RETURNING breaker_tripped plumbing. This same guarded INSERT is the proposed vehicle for the advisory-caps fix.

The Measured Problem

  • Geography: PM on pmv2-zurich (GCP europe-west6-a); Supabase mgmt eu-west-1 (Dublin). ~183ms per sequential round-trip.
  • Deploy measurement (N=196, 3h): gates_ms median 733ms / p90 1179ms / p99 1914ms / max 2750ms; transit_ms median 220ms; book_ms median 212ms.
  • Config reads were the LARGEST controllable stage on the entry path — bigger than transit and book fetch combined.
  • The path did 7 config SELECTs, not 4: place_entry (execute.rs:99) re-resolves mode AFTER the book fetch, outside the gates_ms window. True cost ~1.28s/entry.

The self-reinforcing backlog

levi/gyula (dry, ~12k messages behind head) logged gates_ms == 0 on 28% of entries — the early-return log sites, overwhelmingly stale-gate skips. The backlog is self-reinforcing: being behind makes messages expire, expired messages are dropped, and the drain never catches up. Latency on the entry path feeds directly into this loop.

Design Corrections Found by Adversarial Review — BEFORE Building

These are the value of the note. Each is a trap in the v1 spec that review caught pre-implementation.

  1. The v1 “accepted risk” was unreachable. The spec’s headline accepted risk — 300s of blind live trading during a DB outage — could not happen: fast_reserve’s INSERT and mark_submitted both need the DB, so a dead DB places zero orders and fails closed on its own. The real exposure was the opposite: a healthy DB with a stale cache.
  2. v1’s ArcSwap design was racy by construction. Three concurrent store() writers (poller, refresh, breaker-patch) meant an in-flight load() begun before a /halt committed could land after it and silently restore live for up to 30s on a healthy DB. Fix: ONE writer task owns the sole watch::Sender. That single-writer rule replaced the generation counter and CAS loop entirely.
  3. Boot fail-open trap. An empty first snapshot with loaded_at = Instant::now() reads as FRESH → every wallet resolves Off → SkipAck maps to Ack (consumer.rs:64) → live entries silently dropped and NEVER redelivered, on every restart. Fix: hard-fail the process if the first load() fails.
  4. Stale guard must not trip the breaker. decide3(None, ..) returns breaker_tripped: true, so a naive stale guard would trip the BREAKER gate and spam alerts. The stale path must return eff: Off with breaker_tripped: false.
  5. decide3 hardcodes size_scale: Decimal::ONE and priority: 1. The resolver must copy the real values from WalletCfg afterwards — otherwise every order silently becomes 1× while the existing tests still pass.
  6. clamp_size_scale was applied at read time. Moving reads to a snapshot means clamping at load — otherwise a direct-SQL size_scale = 100 becomes a 100× order.
  7. A torn 4-query load() errs toward live (the unsafe direction). Fix: run it in one REPEATABLE READ transaction.
  8. Alerts must be poller-driven, not read-driven. A read-driven staleness ladder is silent during a traffic lull — exactly when a failing refresh matters most.

Meta-lesson

Corrections 3, 5, and 6 all share a signature: the wrong behavior passes the existing tests. A cache that resolves every wallet to 1×, or reads a raw size_scale=100 as a 100× order, is invisible to the current suite. Adversarial pre-build review — not the test run — is what caught them.

Key architectural idea — move the authority check to a round-trip you already pay

Reusable pattern

When you cache authority, move the authority check to a round-trip you already pay.

fast_reserve’s INSERT became a CTE conditional on the DB’s own mode / breaker_tripped, distinguishing guard-rejection from ON CONFLICT dedup by returning the gate columns alongside the inserted id. Zero extra round-trips, and a stale cache can never place an unauthorized live order — final authority is the DB at insert time, not the cache.

This collapsed the whole design:

  • No generation counter.
  • No breaker snapshot-patch.
  • No RETURNING breaker_tripped plumbing.

The same guarded INSERT is the proposed home for the missing cap gates — see pmv2-live-caps-are-advisory-not-blocking-2026-07-09.

Implementation Constraints Worth Keeping

Do not add arc_swap or moka

Both are absent from Cargo.lock, and Dockerfile:24 builds cargo build --locked — adding either breaks CI. tokio::sync::watch is already available, and Receiver::borrow() is a cheap synchronous read. This is what made the single-writer watch::Sender model (correction #2) the natural fit.

sqlx pool defaults tax every cross-region query

PgPoolOptions defaults test_before_acquire = true (a ping round-trip before EVERY acquire) and min_connections = 0 (idle conns dropped → TLS reconnect). Across the Zurich→Dublin hop that is ~150ms per query, and it taxes the reserve INSERT, mark_submitted, and every ledger write — none of which a config cache removes. Tune these regardless of the cache work.

Status

  • Branch: feat/entry-config-cache @ ec0b666 (10 commits).
  • Tests: 226 lib tests + 295 db-tests green.

Graph View

Backlinks