Levandor

Crypto Emit — EMIT_MAX_FIRES Cap Clear (2026-06-25)

6 min read

Two-step debugging session on 2026-06-25 ~19:25Z to clear a stuck EMIT_MAX_FIRES cap on the apps-shortterm-crypto-algo-emit Pattern B container that had been spinning in paused state for hours after the crypto Pattern-A → Pattern-B cutover. The producer’s first hypothesis (JetStream subject-capture failure) was wrong; the real cause was 50 pre-cutover rows in crypto_shortterm.emitted_signals (Pattern A residue) hydrating an in-process guardrail counter at boot, with EMIT_MAX_FIRES=50 as the compile-time default. Companion to crypto-shortterm-emit-guardrails (BLOCKER-3 architecture context) and reinforces [[gcp-terraform-ansible-gotchas|gotcha #38]] (in-process counters only re-read on boot).

For Agents

Two reusable artifacts came out of this session: (a) a recipe for proving NATS stream subject-capture from inside the nats container without admin NKey CLI access, and (b) a confirmation that the BLOCKER-3 hydration semantics in crypto-shortterm-emit-guardrails cut both ways — they make the emitter restart-safe, but they also mean TRUNCATE alone is insufficient to clear the cap. Bouncing the container is mandatory.

Symptom

apps-shortterm-crypto-algo-emit (Pattern B service, image pmv2-crypto-algo:latest @ sha256:9f88f34f, started 2026-06-24T21:17:54Z) had been spinning every ~1s on:

INFO emit: EMIT_MAX_FIRES reached, signal emission paused max_fires=50

for ~22 hours by 2026-06-25 ~19:25Z. Zero JetStream publish attempts in that window. The producer (cryptoshort agent) flagged it via babylon DM #701.

Diagnosis — two steps

Two steps because the producer’s first hypothesis was wrong.

Step 1 (wrong) — JetStream subject-capture hypothesis

Hypothesis: Crypto suspected JetStream subject-capture failure. The thinking was that the NKey publish-permit looked fine but maybe PMV2_ORDERS stream wasn’t actually capturing pmv2.order.crypto_shortterm_latency_test.> (i.e. the publish was succeeding but landing in the void).

Step 2 (right) — falsify subject-capture, find Pattern A residue

Verify subject-capture from inside nats container using the NATS monitoring port (http: 8222, exposed inside the container at /etc/nats/nats.conf):

docker exec apps-nats-nats-1 \
  wget -qO- 'http://127.0.0.1:8222/jsz?streams=true&config=true&consumers=false'

Result: PMV2_ORDERS stream has subjects: ["pmv2.order.>"] (wildcard). The subject pmv2.order.crypto_shortterm_latency_test.> IS captured by the stream. Hypothesis falsified.

Real cause — three things lined up:

  1. Pattern-A residue: 50 pre-cutover rows in crypto_shortterm.emitted_signals (carried over the Pattern A → B cutover; nobody truncated on switchover).
  2. Compile-time default EMIT_MAX_FIRES=50: the binary DOES read the env var if set, but the apps.yml shortterm-crypto-algo-emit entry omits it AND secrets/crypto-shortterm.env omits it. Default wins.
  3. Boot hydration: per BLOCKER-3, on startup the emitter runs SELECT … FROM emitted_signals and seeds a total_fires counter from the row count. When total_fires >= EMIT_MAX_FIRES, the binary sets paused state and short-circuits BEFORE any JetStream publish. total_fires = 50 and EMIT_MAX_FIRES = 50 → paused at boot → never publishes → never increments the row count → never recovers on its own.

Fix — TRUNCATE + container restart

The cap is held in a process-local counter, hydrated from the DB once at boot. So:

Step A — TRUNCATE the table

TRUNCATE TABLE crypto_shortterm.emitted_signals;

Producer-DB mutation is outside the standing auth scope

Andras’s #480 standing-auth carve-out covers deploys / env / NATS / redispatch / restart — NOT mutations to producer-owned database state. Even with the producer (cryptoshort) giving per-task go-ahead via DM, a TRUNCATE needs explicit operator approval. Captured chain:

  • Crypto authorized via babylon DM #705: “GO with option A — truncate. Authorized.”
  • Operator (Andras) surfaced + approved → executed.

Step B — confirm TRUNCATE alone does NOT clear the cap

After the TRUNCATE landed, we watched the container log for ~40s. It kept spamming EMIT_MAX_FIRES reached, signal emission paused max_fires=50 at ~1Hz. The cap is in-process, not in the DB. The hydration is one-shot at boot.

Step C — bounce the container

sudo systemctl restart apps-shortterm-crypto-algo-emit.service

After restart, the boot log shows:

INFO emit: guardrail hydrated from DB hydrated=0

confirming the reset.

Verification — three signals

Post-restart checks (in priority order):

  1. DB row growth. SELECT count(*) FROM crypto_shortterm.emitted_signals should advance off 0 on the first signal fire.
  2. JetStream sequence. PMV2_ORDERS last_seq from /jsz should advance past 155 on the first publish.
  3. Emit log. Confirms publish + PubAck rather than the EMIT_MAX_FIRES reached spam.

NATS /jsz recipe — proving stream subject-capture

Recipe — JSM stream config from inside the nats container

When you need to prove a stream IS (or ISN’T) capturing a subject, and you don’t have admin NKey CLI access, exec into the nats container and hit the monitoring port directly:

docker exec apps-nats-nats-1 wget -qO- \
  'http://127.0.0.1:8222/jsz?streams=true&config=true&consumers=false'

Returns full JSM stream config including subjects: […]. Looks like:

{
  "streams": [{
    "name": "PMV2_ORDERS",
    "config": {
      "subjects": ["pmv2.order.>"]
    },
    "state": {
      "last_seq": 155,
      "messages": 155
    }
  }]
}

The subjects array on config is the wildcard set the stream binds to — match your publish subject against it. The state.last_seq is the headline signal for “has anyone published recently” — bump it by triggering a publish and re-query.

Why the in-container monitoring port works without auth

The http: 8222 line in /etc/nats/nats.conf exposes the monitoring HTTP endpoints. Inside the container (loopback), no credentials are required. We deliberately do NOT publish :8222 outside the container — see [[gcp-terraform-ansible-gotchas|gotcha #37]] for why (-connz leaks client IPs, -subz leaks subject names).

NKey publish-permit check — cohort_publisher

While investigating, verified that the cohort_publisher NKey UCZLZYBWPK4ENZKA7PMFWS6Z7HCYP77NEVGEKZ4QNRH2VCIW4TIOZUSB (the NKey the crypto emitter uses) in /etc/nats/nats.conf has the expected permits — so the path from publish → PubAck delivery is unobstructed and we can rule out [[gcp-terraform-ansible-gotchas|gotcha #56]]-style silent-fail:

DirectionSubjectPermit
publishpmv2.order.crypto_shortterm_latency_test.>allow
publish$JS.API.>allow (JetStream API surface)
publish$JS.ACK.>allow (PubAck delivery)
subscribe_INBOX.>allow (PubAck reply path)

This was the second check that ruled out NATS as the cause. With subject-capture verified AND publish permits + ACK reply path verified, the only remaining surface was the emitter’s own guardrail — which is what landed us on the Pattern-A-residue hydration.

Approval chain — for future autopilot tuning

The chain that played out for this task is worth recording because it’s exactly the shape future autopilot/auto-act rules need to handle:

  1. Producer (cryptoshort) identifies the issue and proposes Option A (truncate).
  2. Producer per-task go-ahead: babylon DM #705 — “GO with option A — truncate. Authorized.”
  3. Standing-auth carve-out (Andras #480): covers deploys / env / NATS / redispatch / restart. Producer-DB mutation falls outside this carve-out even with producer’s per-task go-ahead.
  4. Operator (Andras) approval surfaced + granted → execute.

For future autopilot tuning

If we widen the standing-auth carve-out to include producer-owned database mutations under a producer’s per-task DM authorization, this exact session could have collapsed from 4 steps to 2. The trade-off: producer agents become a vector for unintended DB writes; the deploy/operator agent loses a final review checkpoint. Worth a deliberate decision in a future babylon protocol round.

Graph View

Backlinks