Levandor

SigNoz Dashboard Housekeeping (2026-06-18) — NATS exporter + host metrics opt-in + dashboard rework

10 min read

What this captures

Round of OTel housekeeping on the polymarket-infra VM (2026-06-18): added a prometheus-nats-exporter sidecar to the nats compose stack, opted four host_metrics scrapers into emission (CPU / FS / memory / paging utilization were silently disabled), and reworked three SigNoz dashboards (new NATS+JetStream board, plus APM log-rate panels and richer host I/O on the existing host board). Four new gotchas captured (#34–#37).

This note has two halves:

  • Part A — Infra changes = uncommitted changes in /Users/levander/levandor/terraform. Need a make configure (or workflow_dispatch on deploy-nats.yml) to roll out. NATS dashboard panels stay empty until then.
  • Part B — SigNoz dashboards = live and applied via the SigNoz HTTP API on 2026-06-18.

Dashboard UUID quick reference

DashboardUUIDStatus
polymarket-infra — NATS + JetStream019eda9d-139f-7526-b22d-c4d8a4a3cdf2NEW (this session)
polymarket-infra — App services (APM)019e6a21-9eee-7e14-9603-1e00ebd59194UPDATED
polymarket-infra — Host & Tailscale019e6a1c-5be5-7383-83ff-23f8562038f4UPDATED

All three live at https://eu2.signoz.cloud/dashboard/<uuid>.

Part A — Infra changes

Uncommitted — roll out with make configure

All five edits below are sitting in the working tree at /Users/levander/levandor/terraform. Until they are applied to the VM via make configure (or gh workflow run deploy-nats.yml), the new NATS dashboard panels render empty and the four host-metric panels added in Part B remain silent.

A1. NATS sidecar — prometheus-nats-exporter

NATS server’s :8222 endpoint returns JSON (/varz, /connz, /jsz, …), not Prometheus text. To feed nats_* metrics into the OTel collector we run prometheus-nats-exporter as a sidecar that translates the JSON endpoints into a Prometheus scrape target. See [[#G4 (gotcha #37). NATS :8222 is JSON, not Prometheus — needs prometheus-nats-exporter|G4]] below.

ansible/roles/nats/templates/compose.yml.j2 gained a third service alongside nats and nats-tailscale:

nats-exporter:
  image: {{ nats_exporter_image }}
  container_name: nats-exporter
  restart: always
  command:
    - "-port=7777"
    - "-varz"
    - "-connz"
    - "-subz"
    - "-routez"
    - "-gatewayz"
    - "-leafz"
    - "-jsz=all"
    - "http://nats:8222"
  ports:
    - "{{ nats_exporter_listen }}:7777"
  depends_on:
    - nats

Key choices:

  • All six topology endpoints enabled (-varz -connz -subz -routez -gatewayz -leafz -jsz=all). Cheap to scrape and they future-proof the dashboard when we add the cohort-publisher subject, more consumers, leaf nodes, etc. -jsz=all is the one that emits the JetStream stream/consumer-level metrics (gnatsd_varz_jetstream_*, gnatsd_consumer_*, nats_jetstream_consumer_*).
  • Connects to http://nats:8222 over the compose-internal docker network. The exporter is in the same compose project so nats is the in-network DNS name. Loopback only on the host side (see next bullet).
  • Bound to 127.0.0.1:7777 on the host. Not exposed publicly. The OTel collector runs native (non-containerized) on the VM, so it scrapes the loopback port like it scrapes [[observability-flow|tailscaled on :5252]]. Do not expose this port — -connz leaks IPs of every NATS client and -subz leaks subject names.

ansible/roles/nats/defaults/main.yml got two new vars:

nats_exporter_image: "natsio/prometheus-nats-exporter:0.17.3"
nats_exporter_listen: "127.0.0.1:7777"

ansible/roles/nats/tasks/main.yml got an image-pull task next to the existing Pull nats server image:

- name: Pull nats-exporter image
  community.docker.docker_image:
    name: "{{ nats_exporter_image }}"
    source: pull

A2. Prometheus scrape job in the OTel collector

ansible/roles/monitoring/templates/config.yaml.j2 receivers.prometheus.config.scrape_configs grew from one entry (tailscaled) to two:

prometheus:
  config:
    scrape_configs:
      - job_name: tailscaled
        scrape_interval: 60s
        metrics_path: /debug/metrics
        static_configs:
          - targets: ['127.0.0.1:5252']
      - job_name: nats
        scrape_interval: 60s
        static_configs:
          - targets: ['127.0.0.1:7777']

Same pattern as tailscaled: localhost target, 60s interval, default metrics_path: /metrics. Already wired into the metrics pipeline because prometheus is in the existing receiver list.

A3. CD workflow smoke check

.github/workflows/deploy-nats.yml smoke check now also asserts the exporter is up. Two additions:

  1. Container presence check expanded from nats nats-tailscale to nats nats-exporter nats-tailscale. Loops over each name; MISSING container: $ctr if docker ps --format doesn’t match.
  2. New EXPORTER_RC step: ssh ops@polymarket-infra "curl -sf http://127.0.0.1:7777/metrics | head -5" — fails the workflow if the exporter isn’t returning anything on /metrics.

Both fold into the final exit-status check alongside COMPOSE_RC / CTR_OK / NAME_RC.

A4. Four host_metrics scrapers explicitly opted in

This is the one that surfaced the silently-broken panels. The OTel host_metrics receiver’s scrapers emit a curated default subset — and the *.utilization metrics in particular are off by default. The CPU and filesystem panels on the host dashboard had been silently rendering empty because system.cpu.utilization and system.filesystem.utilization weren’t being emitted by the collector at all (see [[#G3 (gotcha #36). OTel host_metrics *.utilization metrics are NOT emitted by default|G3]]).

The fix in ansible/roles/monitoring/templates/config.yaml.j2:

host_metrics:
  collection_interval: 60s
  scrapers:
    cpu:
      metrics:
        system.cpu.utilization:
          enabled: true
    load:
    memory:
      metrics:
        system.memory.utilization:
          enabled: true
    disk:
    filesystem:
      metrics:
        system.filesystem.utilization:
          enabled: true
    network:
    paging:
      metrics:
        system.paging.utilization:
          enabled: true

Four scrapers (cpu, memory, filesystem, paging) gained explicit metrics: blocks. Three others (load, disk, network) left bare because their defaults are already correct.

Where the docs hide this

The default-on/off matrix is in each scraper’s documentation.md in the contrib repo (e.g. receiver/hostmetricsreceiver/internal/scraper/cpuscraper/documentation.md). It’s NOT in the receiver’s top-level README. Easy to miss when wiring host metrics for the first time.

A5. Rollout

cd /Users/levander/levandor/terraform
make configure   # site.yml runs all roles; nats + monitoring tags would scope it tighter
# OR
gh workflow run deploy-nats.yml -R wowjeeez/terraform --ref main

After rollout, verify:

ssh ops@polymarket-infra 'curl -sf http://127.0.0.1:7777/metrics | head'   # exporter alive
ssh ops@polymarket-infra 'curl -s :8888/metrics | grep -E "receiver_accepted_metric_points.*prometheus|system_cpu_utilization"' | head  # collector picked them up

Then the new NATS dashboard panels (Part B item B1) should populate within ~60s.

Part B — SigNoz dashboards

Applied via the SigNoz HTTP API on 2026-06-18.

B1. NEW: polymarket-infra — NATS + JetStream

UUID 019eda9d-139f-7526-b22d-c4d8a4a3cdf2. Nine panels, all sourced from the prometheus-nats-exporter scrape:

#PanelSource metric(s)Notes
1scrape-up valueup{job="nats"}green = 1, red = 0; this is the canary for the exporter being healthy
2active connectionsgnatsd_varz_connectionsrough load proxy; expect 1-3 today (wx-signal + pmv2 + cohort publisher)
3subscriptionsgnatsd_varz_subscriptionssanity check — should match the number of consumers + monitoring subscriptions
4slow consumersgnatsd_varz_slow_consumersshould be 0; >0 means downstream is falling behind
5msg-rate in/outrate(gnatsd_varz_in_msgs[1m]), rate(gnatsd_varz_out_msgs[1m])throughput
6byte-rate in/outrate(gnatsd_varz_in_bytes[1m]), rate(gnatsd_varz_out_bytes[1m])bandwidth
7JetStream WEATHER_SIGNALS msgs+bytesgnatsd_jetstream_stream_total_messages{stream_name="WEATHER_SIGNALS"}, ..._bytesstream-level retention check
8JetStream pmv2 consumer backloggnatsd_jetstream_consumer_num_pending, ..._num_ack_pending (filtered to consumer_name="pmv2")the headline lag dashboard — if pending climbs while ack_pending stays steady, the consumer is keeping up; if both climb, the consumer is choking
9pmv2 redeliveries (counter)gnatsd_jetstream_consumer_num_redelivered{consumer_name="pmv2"}>0 means messages are being NAK’d or timing out and getting re-pulled

For Agents

The headline panel here is #8 (pmv2 backlog). Pending vs ack_pending tells you whether the consumer’s CPU is the bottleneck or its downstream (e.g. Polymarket API / Supabase write) is. When triaging “are signals reaching the bot in real time?” — this is the first place to look.

B2. UPDATED: polymarket-infra — App services (APM)

UUID 019e6a21-9eee-7e14-9603-1e00ebd59194. Two new panels added to the existing per-service APM board:

  1. Log severity split (graph) — count of log records per minute, groupBy: [severity_text, service.name]. Catches noise spikes per-service. Severity colors line up with SigNoz conventions: ERROR red, WARN amber, INFO green/cyan, DEBUG/TRACE grey.
  2. ERROR log rate (value panel) — single big number for count(severity_text="ERROR") over the dashboard’s time window, summed across all services. Eyeball / alert candidate.

Existing service-level RED-metric panels (call rate, p99, error rate) untouched.

B3. UPDATED: polymarket-infra — Host & Tailscale

UUID 019e6a1c-5be5-7383-83ff-23f8562038f4. Two clusters of changes:

Added panels (host I/O + Tailscale liveness):

  1. Disk I/O bytes — rate(system.disk.io[1m]) (read/write split). Captures bytes/sec moving in/out of the e2-small’s 20 GB pd-balanced volume.
  2. Disk IOPS — rate(system.disk.operations[1m]). Useful alongside (1) to spot read-amplification or small-IO storms.
  3. Swap paging ops — rate(system.paging.operations[1m]) (in/out split). e2-small has 2 GB RAM and any swap activity is a yellow flag worth investigating.
  4. Network errors+drops — rate(system.network.errors[1m]) + rate(system.network.dropped[1m]). Both should be flat-zero in steady state; any nonzero is interesting.
  5. tailscaled uptime (value panel) — derived from the process_start_time_seconds metric on the tailscaled scrape. Numeric uptime in the corner is a quick “is the daemon still the same process I think it is” check.

Scoped existing panels to host.name=polymarket-infra:

The tailscale-internal metrics (goroutines, DNS metrics dns_*, DERP metrics derp_*) were previously unfiltered. Now they’re all filtered to host.name=polymarket-infra so that if we add a second VM, this dashboard stays scoped to the primary infra host and the other VM gets its own board. Same pattern: explicit resource-attribute filter at the panel level rather than relying on the dashboard-level scope.

Gotchas (added to the master list)

Four new entries appended to gcp-terraform-ansible-gotchas (numbered #34–#37; total now 37). Full text lives in that document — summaries here:

G1 (gotcha #34). SigNoz create_dashboard HTML-escapes & in titles

The SigNoz MCP create_dashboard mutation HTML-escapes ampersands in the dashboard title before persisting. So a title like NATS & JetStream is stored as NATS &amp; JetStream and renders that way in the UI’s sidebar. The NATS dashboard’s title was created as polymarket-infra — NATS + JetStream (using +) to dodge this. The already-existing App services (APM) and Host & Tailscale boards were created before this was known — the latter’s title was corrected via update_dashboard after observing the escaped form.

Operating rule: Prefer +, and, or , over & when creating SigNoz dashboards via API. If an & slips through, update_dashboard rewrites the field correctly (no double-escape).

G2 (gotcha #35). SigNoz host metrics are stored as dotted names but legacy widgets reference underscore form

SigNoz stores host metrics under the dotted canonical form that OTel emits (system.memory.usage, system.cpu.utilization). However, older dashboards and the Metric Explorer auto-complete still surface the underscore-substituted form (system_memory_usage). Both work for queries — SigNoz normalizes — but they read as if they’re different metrics.

Operating rule: When adding new widgets, use the dotted canonical form (system.memory.usage). Match what OTel emits, match what the docs cite. Leave existing underscore-form references in legacy widgets alone (no functional difference, churn risk).

G3 (gotcha #36). OTel host_metrics *.utilization metrics are NOT emitted by default

The OTel host_metrics receiver’s cpu, memory, filesystem, and paging scrapers default to NOT emitting their system.*.utilization metrics — only the raw *.usage / *.time counters. To get system.cpu.utilization (and FS / mem / paging utilization) the per-scraper metrics: block must explicitly opt in:

cpu:
  metrics:
    system.cpu.utilization:
      enabled: true

This bit our host dashboard silently. CPU and FS utilization panels had been empty since Spec 1 deployment and nobody noticed because the *.usage metrics on adjacent panels populated fine. The default-on/off matrix is in each scraper’s documentation.md in opentelemetry-collector-contrib/receiver/hostmetricsreceiver/internal/scraper/*/, not in the receiver-level README — easy to miss.

Operating rule: When wiring a SigNoz host-resources dashboard, audit which metrics it queries against the contrib scraper docs and explicitly enable any non-default metric in the collector config. Don’t trust that “the metric exists in the SDK” means “the collector is shipping it.”

G4 (gotcha #37). NATS :8222 is JSON, not Prometheus — needs prometheus-nats-exporter

NATS server’s monitoring port (:8222) returns JSON at /varz, /connz, /subz, /routez, /gatewayz, /leafz, /jsz. It is NOT a Prometheus text endpoint. Pointing the OTel collector’s prometheus receiver directly at 127.0.0.1:8222 will fail to parse.

The translation layer is prometheus-nats-exporter (image natsio/prometheus-nats-exporter:0.17.3). It connects to NATS over HTTP, polls the JSON endpoints (enabled per -varz -connz -subz -routez -gatewayz -leafz -jsz=all flags), and exposes Prometheus-format metrics on its own port (here :7777).

Operating rule for this VM: the exporter binds to 127.0.0.1:7777 on the host (loopback only) and the native OTel collector scrapes it like it scrapes [[observability-flow|tailscaled on :5252]]. Do not expose :7777 publicly — -connz leaks every client IP and -subz leaks subject names. JetStream metrics specifically require -jsz=all; without it the stream/consumer-level series are absent.

Files touched

Infra (uncommitted, /Users/levander/levandor/terraform):

  • ansible/roles/nats/templates/compose.yml.j2 — new nats-exporter service.
  • ansible/roles/nats/defaults/main.yml — nats_exporter_image, nats_exporter_listen vars.
  • ansible/roles/nats/tasks/main.yml — Pull nats-exporter image task.
  • ansible/roles/monitoring/templates/config.yaml.j2 — nats Prometheus scrape job + four *.utilization opt-ins.
  • .github/workflows/deploy-nats.yml — smoke check expects nats-exporter container + curls :7777/metrics.

Vault:

  • This note: projects/levandor-infra/signoz-dashboard-housekeeping-2026-06-18.md.
  • gcp-terraform-ansible-gotchas — appended gotchas #34–#37.
  • observability-flow — extended prometheus scrape config block + Tailscale-style sidecar note.
  • levandor-infra — overview tweak (3 dashboards) + gotcha count bump.
  • LOG — entry under 2026-06-18.
  • TOPICS — links added under Telemetry & Monitoring + Gotchas & Learnings + a new SigNoz Dashboards H2.

Graph View

Backlinks