Emit Producer-Side Latency — the DB Claim INSERT Gated the NATS Publish (publish-first + DLQ)

Measured a producer-side latency leak in emit: the synchronous emitted_signals claim INSERT to Supabase sat on the critical path, before the NATS Directive publish, costing 234–301ms median / p90 ~1–2.5s / max ~3s on every fire — the emit-side complement to PM’s consumer-side R0–R6 work. Fix (feat/emit-publish-first-dlq @ 6cd9487, pushed, not rolled) is a flag-gated publish-first + async-record + DLQ path. The catch: publish-first surrenders the DB claim’s cross-process dedup role (the whole point of the C1 persist-before-publish invariant) and is safe to flip only if JetStream message-dedup is confirmed on the crypto entry stream.

For Agents — binding facts

1. Measured leak: crates/observe/src/bin/emit.rs does a synchronous INSERT INTO crypto_shortterm.emitted_signals … ON CONFLICT DO NOTHING (the claim, already logged as claim_ms) and only calls publish_entry (the NATS Directive) after it commits. That claim measured 234–301ms median, p90 ~1–2.5s, max ~3s across live lanes (deploy grepped the logs). So a quarter-second (tail up to 3s) sits between “signal ready” and “Directive out” on every fire.
2. Cause = cross-region. emit runs on pmv2-zurich (GCP europe-west6, Zurich); Supabase project mkofmdtdldxgmmolxxhc is eu-west-1 (Dublin) → cross-region round-trip + TLS/pooler/query overhead. Plausible real contributor to the adverse-fill loss, since 5m adverse selection happens in the final seconds.
3. Fix: branch feat/emit-publish-first-dlq @ 6cd9487 (pushed, NOT rolled), behind env flag EMIT_PUBLISH_FIRST (default OFF = byte-identical legacy C1 path). ON = publish first, record async, DLQ on failure.
4. Safety linchpin (OPEN): publish-first drops the C1 DB-claim cross-process dedup. Do NOT flip until JetStream message-dedup with a duplicate window ≥ ~2 min is CONFIRMED on the crypto entry stream (pending PM answer, babylon pmv2 #1160). Otherwise a deploy-roll container overlap could double-fire.
5. Rollout: dark (flag unset) → 1-lane canary → fan out. Set EMIT_DLQ_PATH to a persistent volume (default /tmp is tmpfs on many containers → lost on restart; loss = one analytics row, never a trade).

Root cause — the claim INSERT gates the publish

The live emit path (the C1 persist-before-publish ordering) is, per fire:

1. Synchronous INSERT INTO crypto_shortterm.emitted_signals … ON CONFLICT DO NOTHING — the claim (dedup authority + spend accounting).
2. Only after that commits, call publish_entry → the NATS Directive goes out.

The claim latency is already instrumented in emit.rs as claim_ms. Deploy grepped the live logs across lanes:

Metric	Claim latency
median (p50)	234–301ms
p90	~1–2.5s
max	~3s

Why it is this slow

emit runs on pmv2-zurich (GCP europe-west6, Zurich) but the Supabase project mkofmdtdldxgmmolxxhc lives in eu-west-1 (Dublin). Every claim is a cross-region DB round-trip plus TLS handshake, connection-pooler, and query overhead. That whole cost is paid synchronously, before the Directive can leave the box.

Why this plausibly costs money

The latency budget put the publish→fill target at ~128ms (p90 adverse drift), <50ms strongly recommended. A 234–301ms median claim (tail to ~3s) sitting ahead of the publish alone blows that budget before the Directive is even on the wire. In the 5m market, adverse selection happens in the final seconds — extra producer latency walks every fire deeper into the ask-reprice window, exactly the mechanism behind the ask-runs-away losses.

Producer-side complement to PM’s R0–R6

This is the producer-side half of the latency story. PM’s consumer-side telemetry harness (R0–R6, measure-before-mitigating) already showed the dominant live cost is pre-build latency (emit→signal gap p50 ~2.3s, first-retry-attempt p50 ~4–4.5s / p90 ~8s) and that ~50% of live buys fail price-moved-past-limit. The claim_ms leak is a concrete, fixable chunk of that emit→signal gap on the crypto side of the seam. Same discipline applies: this was measured (deploy log grep), not assumed.

The fix — publish-first + async record + DLQ

Branch feat/emit-publish-first-dlq @ 6cd9487 (pushed, NOT rolled). Entirely behind env flag EMIT_PUBLISH_FIRST:

• EMIT_PUBLISH_FIRST unset / off (default): byte-identical to the legacy C1 claim-then-publish path. Zero behavioural change when dark.
• EMIT_PUBLISH_FIRST=true: publish the Directive FIRST, then record emitted_signals asynchronously (tokio::spawn) off the critical path. The claim round-trip no longer gates the publish.

Async record + Dead-Letter Queue

When the async insert fails, the row is not lost:

• Append the record to a JSONL DLQ file (EMIT_DLQ_PATH, default /tmp/emit_dlq_<label>.jsonl).
• Fire a rate-limited (1 / 5 min) Telegram alert via telegram.outbound.
• A drainer retries the DLQ every 60s (same ON CONFLICT DO NOTHING) and atomically rewrites the survivors back to the file.

New module crates/observe/src/dlq.rs: append / read_all / atomic-rewrite, skips corrupt lines on read, 4 TDD tests. Shared helpers insert_signal + emit_directive keep the legacy and publish-first paths DRY (one implementation of each, not a fork). A shared async mutex dlq_lock serializes DLQ file access so the drainer’s rewrite can’t clobber a concurrent append.

Legacy (C1) vs publish-first fire flow

flowchart TD
    Sig["Signal ready<br/>(regime ∧ select_side)"] --> Flag{"EMIT_PUBLISH_FIRST?"}
    Flag -->|"off — default (legacy C1)"| Claim["SYNC claim INSERT<br/>emitted_signals<br/><i>claim_ms 234–301ms,<br/>p90 1–2.5s, max ~3s</i>"]
    Claim --> PubL["publish Directive<br/><b>(gated behind the claim)</b>"]
    Flag -->|"on — publish-first"| PubF["publish Directive FIRST<br/><b>(off the DB critical path)</b>"]
    PubF --> Async["tokio::spawn:<br/>async insert_signal"]
    Async --> Ok{"insert ok?"}
    Ok -->|yes| Done["recorded"]
    Ok -->|no| Dlq["append DLQ (JSONL)<br/>+ rate-limited TG alert<br/>→ drainer retries 60s,<br/>atomic-rewrite survivors"]
    style Claim fill:#3d2020,stroke:#a33,color:#fff
    style PubL fill:#3d2020,stroke:#a33,color:#fff
    style PubF fill:#264653,stroke:#2a9d8f,color:#fff
    style Async fill:#264653,stroke:#2a9d8f,color:#fff
    style Dlq fill:#2d2d2d,stroke:#888,color:#fff
    style Done fill:#2d2d2d,stroke:#888,color:#fff

Safety linchpin — publish-first surrenders the C1 dedup authority

This is the whole risk, and it is open.

Do NOT flip EMIT_PUBLISH_FIRST without confirming JetStream dedup

The C1 persist-before-publish invariant deliberately made the DB row the cross-process dedup authority — the claim commits before the publish precisely so two processes (e.g. an old + new container during a deploy roll) can never both fire the same window. Publish-first removes that guarantee.

Publish-first now relies on two weaker mechanisms:

• (a) in-memory fired_windows — per-instance only (does NOT span a deploy-roll container overlap), and
• (b) the Nats-Msg-Id header (= order id source|window, deterministic per window) → JetStream stream message-dedup.

Publish-first is safe to flip only if JetStream message-dedup is enabled on the crypto entry stream with a duplicate window ≥ ~2 min (long enough to cover a deploy-roll container overlap). If dedup is off or the window is too small, publish-first can double-fire during a roll — a real-money duplicate.

The dedup-config question is out to PM on babylon pmv2 #1160. The known wire (from the handover): id = Nats-Msg-Id = crypto_shortterm_latency_test|<window_id>, published to subject pmv2.order.crypto_shortterm_latency_test.entry, consumed by @positionmanager on stream PMV2_ORDERS. The open item is whether that stream has duplicate-window dedup configured at ≥ ~2 min.

The three-chapter ordering saga

This note is the third turn of the emit publish-ordering decision, and it is a deliberate partial reversal:

Chapter	Ordering	Guarantee bought	Cost accepted
bc38734 (crypto-shortterm-emit-guardrails)	publish → then mark fired	never lose a signal (retry on publish fail)	duplicate spend risk on crash/roll
C1 feat/down-side 2026-06-28 (crypto-shortterm-down-side-money-path-fixes-2026-06-28)	claim (DB) → then publish	never duplicate real money (DB = dedup authority)	a post-claim publish fail = detectable missed trade + the claim_ms latency this note measures
feat/emit-publish-first-dlq 2026-07-09 (this note)	publish → then async-record (DLQ)	latency (claim off the critical path)	dedup authority moves from DB → JetStream (must be confirmed)

C1 traded latency for dedup safety; publish-first trades it back, and re-establishes dedup at the JetStream layer instead of the DB. That is why the JetStream-dedup confirmation is load-bearing — it is literally re-buying the guarantee C1 gave up.

Rollout — gated on the dedup confirm + Andras

Gated on both the JetStream-dedup confirmation (#1160) and Andras:

1. Roll dark — ship with EMIT_PUBLISH_FIRST unset (legacy path, zero change) so the branch is deployable before the flip is authorized.
2. Canary one lane — set EMIT_PUBLISH_FIRST=true on a single lane; watch pub_lag_ms drop, no double-fire, DLQ empty.
3. Fan out — enable across lanes once the canary is clean.

Op note — the DLQ path must be persistent

EMIT_DLQ_PATH defaults to /tmp/emit_dlq_<label>.jsonl, but /tmp is tmpfs on many containers → the DLQ is lost on restart. Point EMIT_DLQ_PATH at a persistent volume for cross-restart durability. Impact of a lost DLQ entry is only a missing analytics row, never a trade (the trade already published) — but persistence keeps emitted_signals complete for P&L/attribution.

Related

• crypto-shortterm-down-side-money-path-fixes-2026-06-28 — the C1 persist-before-publish invariant this reverses; C1 made the DB claim the cross-process dedup authority, which is exactly the claim_ms cost measured here.
• crypto-shortterm-emit-guardrails — the original bc38734 publish-then-mark ordering (chapter 1 of the saga); C1 superseded it, this note re-inverts it.
• crypto-shortterm-phase1-latency-budget — the ~128ms / <50ms publish→fill budget the claim_ms leak blows through before the Directive is even on the wire.
• crypto-shortterm-lane-state-adverse-fill-2026-07-05 — adverse fill is an execution/latency disease (final-seconds ask reprice); producer latency feeds it.
• crypto-shortterm-fak-nomatch-root-cause-2026-07-02 — the ask-runs-away-within-~0.7s mechanism that any extra producer latency worsens.
• pmv2-measure-before-mitigating-priority-tiers-2026-07-09 — PM’s consumer-side R0–R6 latency-decomposition harness; this is its producer-side complement (and the same measure-before-mitigating discipline).
• crypto-shortterm-phase1-emit — the emit binary + Directive/EntryOrder path being modified.
• crypto-shortterm-handover-2026-06-26 — the live wire (source, Nats-Msg-Id = source|window, subject, stream PMV2_ORDERS).
• crypto-shortterm — project index.