Levandor

pmv2 Fleet — Infra, Observability & NATS Conventions

4 min read

Fleet-wide reference (from @deploy, babylon #pmv2/#general msg #313, 2026-06-22) for observability, logging discipline, and NATS/JetStream conventions. Applies to ALL pmv2 services — cohort_algo, position_manager, weather, crypto, telegram_connector. Components live at /Users/levander/coding/pmv2.

For Agents

This is the canonical fleet contract for telemetry + messaging. Any new pmv2 service MUST follow these rules. When wiring a Rust service, init tracing + tracing-opentelemetry + opentelemetry-otlp pointing at the local OTel collector (never SigNoz directly), and always set Nats-Msg-Id on every JetStream publish.

Observability — OTel → local collector → SigNoz

Never ship directly to SigNoz

All three signals (traces, logs, metrics) go to SigNoz Cloud via the local OTel collector on host polymarket-infra. Never export directly to SigNoz.

Collector endpoints:

  • OTLP gRPC: 127.0.0.1:4317
  • OTLP HTTP: 127.0.0.1:4318
  • From Docker: http://host.docker.internal:4318

Standard env:

OTEL_SERVICE_NAME=<pmv2-service-name>
OTEL_EXPORTER_OTLP_ENDPOINT=<collector endpoint>
OTEL_EXPORTER_OTLP_PROTOCOL=grpc | http/protobuf
OTEL_RESOURCE_ATTRIBUTES=service.namespace=pmv2,deployment.environment=prod

Containerized apps (deployed via the ansible apps role) get these auto-injected. Native / new repos opt in via SDK init.

Logging Discipline

LevelUse forNotes
ERRORoperator-visible failures
WARNnon-fatal anomaliesrate-limit / retry / dedup-skip
INFOstate transitions onlyused sparingly, metered — never per-message above ~1/s
DEBUGhigh-volume per-message behaviorbehind RUST_LOG
  • Log key fields as attributes, not embedded strings: source, source_id, signal_id, order_id, condition_id, outcome_index, dedup_key, subject.

Never log secrets or PII

Never log private keys, NKey seeds, bot tokens, or DB URLs. Log identity (e.g. the NKey pubkey), never the seed.

NATS / JetStream Conventions

Always set Nats-Msg-Id on publish

Dedup happens in-stream within duplicate_window (currently 5 min on all three streams). Set Nats-Msg-Id on every publish.

Streams:

StreamSubjectsRetentionNotes
TELEGRAM_OUTBOUNDtelegram.outbound1dsee telegram-connector-outbound-design
OPSops.commands, ops.results7dcontrol plane needs audit
PMV2_ORDERSpmv2.order.<source>.<entry|exit>—pending PM consumer
WEATHER_SIGNALSlegacy weather.signals, cohort.signals, weather.shadow—until polymarket_fetch retires
  • Durable consumers: each consuming service creates its own with a stable name (e.g. tg_outbound). Producers don’t create streams — @deploy owns stream provisioning.

GOTCHA: Request/Reply on Shared Stream (Dedup Collision)

Silent Dedup Drop on Shared Request/Reply Streams

Discovered 2026-06-24, first live ops round-trip. When a single JetStream stream carries both sides of a request/reply pattern (or any two related messages), the two MUST use distinct Nats-Msg-Id values, or one silently deduplicates the other within the duplicate_window.

What happened: OPS stream carries both ops.commands (published by telegram_connector) and ops.results (published by position_manager). Both used Nats-Msg-Id = command_id. When position_manager published the result with the same msg-id as the inbound command, JetStream silently dropped the result message — the publish call returned success, no error was logged, but the stream sequence never advanced. Symptom: operator’s /status request timed out with no error anywhere, because the result never made it into the stream.

The rule: For request/reply over a shared stream, result msg-ids MUST be distinct from command msg-ids. Correlation lives in the message body (e.g., OpsResult.command_id), not the Nats-Msg-Id.

Fix applied (position_manager 4d27a5f): Result Nats-Msg-Id = <command_id>-result (distinct), correlation via body field OpsResult.command_id. Also upgraded the result publish to JetStream (+ PubAck) so future drops are observable (the ack won’t arrive if the stream rejects the message).

Connector implication: telegram_connector’s tg_ops_results consumer already correlates on the body OpsResult.command_id, never on Nats-Msg-Id — the fix is transparent. Also: a late or duplicate result whose command_id has no pending waiter (e.g., original command timed out) is logged + acked + dropped cleanly.

Tracing

  • One trace span tree per end-to-end signal lifecycle.
  • Span names = function role, not data. Attach data as attributes.
  • Rust stack: tracing + tracing-opentelemetry + opentelemetry-otlp.

Health Checks — Golden Rules

  1. docker ps / systemctl is-active is ground truth.
  2. journald (journalctl -u <svc> -f) is the universal fallback.
  3. SigNoz is the second look.
  4. NATS health via nats stream info / nats consumer info with the pmv2 admin seed (use sparingly).

SigNoz dashboards available today:

  • polymarket-infra — Host & Tailscale
  • polymarket-infra — App services (APM)
  • polymarket-infra — NATS + JetStream

Coordination State (babylon #pmv2)

  • #310 — pmv2-contracts confirmed as a standalone repo (Andras).
  • #311 — inbound ops.* contract still open with @positionmanager.

Graph View

Backlinks