Levandor

Observability Flow — Logs, Traces, Metrics, Tailscale

9 min read

Spec 2b is live (2026-05-27)

Single OTel collector on the polymarket-infra VM carries four independent signal sources to SigNoz Cloud EU2. Apps emit logs+traces over OTLP gRPC with zero per-app config — the apps Ansible role injects standard OTEL_* env vars and host.docker.internal:host-gateway automatically.

Architecture

                          ┌──────────────────────────────┐
                          │  polymarket-infra VM         │
                          │  europe-west3-a, e2-small    │
                          │                              │
   apps in containers ──→ │  host.docker.internal:4317   │ (OTLP gRPC, in-cluster)
   (tracing+OTel SDK)     │  host.docker.internal:4318   │ (OTLP HTTP, in-cluster)
                          │           │                  │
                          │           ▼                  │
   tailscaled  ←─────────→│  127.0.0.1:5252/debug/metrics│ (Prometheus scrape)
   (Tailscale daemon)     │           │                  │
                          │           ▼                  │
                          │  ┌────────────────────────┐  │
                          │  │  otelcol-contrib       │  │
                          │  │  v0.152.1, systemd     │  │
                          │  │  /etc/otelcol-contrib/ │  │
                          │  │     config.yaml        │  │
                          │  └────────────────────────┘  │
                          │           │                  │
                          │           │ + host_metrics   │
                          │           │   (built-in)     │
                          │           │                  │
                          │           ▼                  │
                          │  resourcedetection [system,  │
                          │   gcp] → batch processors    │
                          │           │                  │
                          └───────────┼──────────────────┘
                                      │
                                      ▼ OTLP gRPC + signoz-ingestion-key header
                       ingest.eu2.signoz.cloud:443  (TLS, port 443)
                                      │
                                      ▼
                              ┌─────────────────┐
                              │  SigNoz Cloud   │
                              │  EU2 (Frankfurt)│
                              └─────────────────┘

Three signals + their sources

SignalSourceReceiver in collectorPipeline
Metrics — hostOTel host_metrics (CPU, memory, disk, network, paging, load, filesystem; *.utilization series explicitly opted in since 2026-06-18 — see [[gcp-terraform-ansible-gotchas#36. OTel host_metrics *.utilization metrics are NOT emitted by default|gotcha #36]])host_metrics (built-in, 60s collection_interval)metrics
Metrics — Tailscaletailscaled --debug=127.0.0.1:5252 (Prometheus format at /debug/metrics)prometheus scraping 127.0.0.1:5252 every 60smetrics
Metrics — NATS (since 2026-06-18)prometheus-nats-exporter:0.17.3 sidecar in the [[signoz-dashboard-housekeeping-2026-06-18#A1. NATS sidecar — prometheus-nats-exporter|nats compose project]] translates NATS’s JSON :8222 endpoint into Prometheus format, exposes on 127.0.0.1:7777 (flags: -varz -connz -subz -routez -gatewayz -leafz -jsz=all). NATS :8222 is JSON, not Prom — see [[gcp-terraform-ansible-gotchas#37. NATS :8222 is JSON, not Prometheus — needs prometheus-nats-exporter|gotcha #37]]prometheus scraping 127.0.0.1:7777 every 60smetrics
TracesApp SDK (tracing-opentelemetry for Rust) over OTLP/gRPCotlp.grpc (0.0.0.0:4317)traces
LogsApp SDK (opentelemetry-appender-tracing bridge for Rust) over OTLP/gRPCotlp.grpc (0.0.0.0:4317)logs

All five end up at the same otlp_grpc exporter pointing at ingest.eu2.signoz.cloud:443 with signoz-ingestion-key header from secrets/signoz_ingestion_key.

Files to read

  • Collector config: ansible/roles/monitoring/templates/config.yaml.j2
  • Collector role: ansible/roles/monitoring/tasks/main.yml (installs otelcol-contrib, configures tailscaled --debug flag, writes config)
  • Compose template (injects OTEL_* env vars for every container): ansible/roles/apps/templates/docker-compose.yml.j2
  • Apps role defaults: ansible/roles/apps/defaults/main.yml (apps_environment: prod)
  • Rust app integration example: crates/polymarket-fetch/src/observability.rs in wowjeeez/polymarket-fetch

Auto-injection: every container gets these env vars

The apps role’s compose template renders this block on every entry by default (opt out per entry with telemetry: false):

environment:
  OTEL_SERVICE_NAME: "{{ item.name }}"
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://host.docker.internal:4317"
  OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
  OTEL_RESOURCE_ATTRIBUTES: "deployment.environment={{ apps_environment }},service.namespace=apps"
extra_hosts:
  - "host.docker.internal:host-gateway"

host.docker.internal:host-gateway is Docker’s built-in alias for the host’s bridge gateway IP (typically 172.17.0.1 on Linux). Container reaches the host’s :4317 via that gateway — no port binding from outside the VM required.

Onboarding a new app — Rust (tracing + tracing-opentelemetry)

The auto-injected env vars are honored by any standard OTel SDK. For Rust with tracing + tracing-opentelemetry + opentelemetry-appender-tracing, drop this single file in the crate and call observability::init(...) from main. All boilerplate, zero per-app config.

Cargo.toml

Workspace deps (verified compatible 2026-05-27):

opentelemetry = "0.28"
opentelemetry_sdk = { version = "0.28", features = ["rt-tokio"] }
opentelemetry-otlp = { version = "0.28", default-features = false, features = ["grpc-tonic", "trace", "logs"] }
opentelemetry-appender-tracing = "0.28"
tracing-opentelemetry = "0.29"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json", "fmt"] }

Version pin matters

The OTel-Rust ecosystem renamed APIs in 0.28 (SdkLoggerProvider/SdkTracerProvider/Resource::builder()). Stick with the 0.28-line. If you bump, expect breakage. (gcp-terraform-ansible-gotchas item to add when it bites.)

src/observability.rs

use std::time::Duration;
 
use anyhow::{Context, Result};
use opentelemetry::{global, trace::TracerProvider as _};
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{
    Resource,
    logs::SdkLoggerProvider,
    propagation::TraceContextPropagator,
    trace::SdkTracerProvider,
};
use tracing_subscriber::{EnvFilter, Layer, layer::SubscriberExt, util::SubscriberInitExt};
 
const TRACER_NAME: &str = "my-app";
const DEFAULT_SERVICE_NAME: &str = "my-app";
const OTLP_EXPORT_TIMEOUT: Duration = Duration::from_secs(5);
 
#[derive(Default)]
pub struct Guard {
    tracer_provider: Option<SdkTracerProvider>,
    logger_provider: Option<SdkLoggerProvider>,
}
 
impl Drop for Guard {
    fn drop(&mut self) {
        if let Some(tp) = self.tracer_provider.take() {
            let _ = tp.shutdown();
        }
        if let Some(lp) = self.logger_provider.take() {
            let _ = lp.shutdown();
        }
    }
}
 
pub fn init(default_filter: &str, json_logs: bool) -> Result<Guard> {
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new(default_filter));
 
    let registry = tracing_subscriber::registry().with(filter);
 
    let fmt_layer = if json_logs {
        tracing_subscriber::fmt::layer().with_writer(std::io::stderr).json().boxed()
    } else {
        tracing_subscriber::fmt::layer().with_writer(std::io::stderr).boxed()
    };
 
    // OTEL_EXPORTER_OTLP_ENDPOINT is set by the apps Ansible role on the VM.
    // When absent (local cargo run), we fall back to stderr-only — same UX as before.
    let Some(endpoint) = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").ok() else {
        registry.with(fmt_layer).init();
        return Ok(Guard::default());
    };
 
    global::set_text_map_propagator(TraceContextPropagator::new());
 
    let service_name = std::env::var("OTEL_SERVICE_NAME")
        .unwrap_or_else(|_| DEFAULT_SERVICE_NAME.to_string());
    let resource = Resource::builder().with_service_name(service_name).build();
 
    let span_exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_tonic().with_endpoint(&endpoint).with_timeout(OTLP_EXPORT_TIMEOUT)
        .build().context("build OTLP span exporter")?;
    let tracer_provider = SdkTracerProvider::builder()
        .with_batch_exporter(span_exporter).with_resource(resource.clone()).build();
    global::set_tracer_provider(tracer_provider.clone());
    let tracer = tracer_provider.tracer(TRACER_NAME);
    let span_layer = tracing_opentelemetry::layer().with_tracer(tracer);
 
    let log_exporter = opentelemetry_otlp::LogExporter::builder()
        .with_tonic().with_endpoint(&endpoint).with_timeout(OTLP_EXPORT_TIMEOUT)
        .build().context("build OTLP log exporter")?;
    let logger_provider = SdkLoggerProvider::builder()
        .with_batch_exporter(log_exporter).with_resource(resource).build();
    let log_bridge = OpenTelemetryTracingBridge::new(&logger_provider);
 
    registry.with(fmt_layer).with(span_layer).with(log_bridge).init();
 
    Ok(Guard { tracer_provider: Some(tracer_provider), logger_provider: Some(logger_provider) })
}

Wire-up in main

#[tokio::main]
async fn main() -> Result<()> {
    let _guard = observability::init("info,my_app=info", false)?;
    // ... rest of main; spans + tracing macros now ship to SigNoz automatically
    Ok(())
}

Hold the Guard until end-of-main so the batch exporters flush on shutdown.

Onboarding non-Rust apps

The OTel SDK in any language reads the standard OTEL_* env vars. Concretely for the most common stacks:

  • Go: go.opentelemetry.io/otel/sdk + otlptracegrpc and otlploggrpc exporters with WithEndpoint(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")) (or use the SDK auto-configure path which reads env directly).
  • Python: opentelemetry-sdk + opentelemetry-exporter-otlp-proto-grpc + opentelemetry-instrumentation for auto-instrumentation. Env vars are auto-honored.
  • Node: @opentelemetry/sdk-node with @opentelemetry/auto-instrumentations-node. Same env vars.

Pattern is identical across languages: the apps role sets the env vars, the SDK reads them, signals flow. No per-app collector endpoint config in code.

Verifying signals are flowing

From the VM (collector-side)

The collector exposes its own metrics on :8888/metrics. Look for paired counters — incoming accepted should match outgoing sent, with zero send_failed:

ssh ops@polymarket-infra curl -s http://127.0.0.1:8888/metrics | grep -E \
  '^otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)_(log_records|metric_points|spans)\b'

Healthy steady-state example (post Spec 2b deploy, ~30 min):

otelcol_receiver_accepted_log_records{receiver="otlp",transport="grpc"} 41
otelcol_exporter_sent_log_records{exporter="otlp_grpc",server_address="ingest.eu2.signoz.cloud",server_port="443"} 41
otelcol_receiver_accepted_metric_points{receiver="host_metrics",transport=""} 33765
otelcol_exporter_sent_metric_points{exporter="otlp_grpc",server_address="ingest.eu2.signoz.cloud",server_port="443"} 33765
otelcol_receiver_accepted_spans{receiver="otlp",transport="grpc"} 143
otelcol_exporter_sent_spans{exporter="otlp_grpc",server_address="ingest.eu2.signoz.cloud",server_port="443"} 143

accepted == sent, zero send_failed lines → end-to-end clean.

See also gotcha 9 for the canonical signal pattern.

From SigNoz Cloud (consumer-side)

  • Services view: https://eu2.signoz.cloud/services — lists every service emitting traces with call rate, p99, error rate. Right now: polymarket-fetch-bot, polymarket-fetch-wallet-monitor, pm-arb-validate, polymarket-fetch-scheduled-run (when its timer fires).
  • Logs view: https://eu2.signoz.cloud/logs — filter service.name=polymarket-fetch-bot (or any OTEL_SERVICE_NAME value).
  • Metrics view: https://eu2.signoz.cloud/metrics-explorer — query system_* (host) and controlclient_*/derp_*/dns_*/magicsock_*/goroutines (tailscaled).
  • Dashboards: https://eu2.signoz.cloud/dashboard — three production boards (also see signoz-dashboard-housekeeping-2026-06-18):
    • polymarket-infra — Host & Tailscale — 019e6a1c-5be5-7383-83ff-23f8562038f4
    • polymarket-infra — App services (APM) — 019e6a21-9eee-7e14-9603-1e00ebd59194
    • polymarket-infra — NATS + JetStream — 019eda9d-139f-7526-b22d-c4d8a4a3cdf2 (added 2026-06-18; populated once prometheus-nats-exporter ships via make configure)

What’s NOT shipped to SigNoz

  • Per-container resource metrics (docker stats). The collector doesn’t run a docker_stats receiver. If we want this, add the docker_stats receiver — requires giving the collector access to /var/run/docker.sock. Not done because host_metrics already covers aggregate resource pressure.
  • Supabase / Polymarket API outbound metrics. The apps’ own traces capture HTTP/Postgres outbound as spans, but if we ever want per-host RED metrics for outbound APIs, we’d add a httpcheck or sqlserverreceiver-style receiver (no Postgres exporter currently configured — see Supabase metrics path below).
  • Cardholder/PII. Per OTel semantic conventions, OTEL_RESOURCE_ATTRIBUTES is deployment.environment + service.namespace only. No user IDs, no auth tokens — those stay in app-side trace attributes and are filtered by the SDK before export.

Supabase metrics path

Supabase Free tier: no native external metric export

Supabase doesn’t expose Prometheus or OTLP for the database engine on free/pro tiers. The dashboard UI at https://supabase.com/dashboard/project/mkofmdtdldxgmmolxxhc/database/reports has CPU/memory/connections charts but no pull-based external feed.

Available paths if you ever want pg_stat_ shipped to SigNoz:*

  1. postgres_exporter container — a Prometheus exporter that scrapes pg_stat_* views via the pooler URL. Run as a service-kind apps entry, add a prometheus receiver to the collector that scrapes its :9187/metrics. ~30 min of setup. Adds another moving piece + an SQL role with pg_monitor granted.
  2. OTel Collector postgresql receiver — postgresqlreceiver in otelcol-contrib. Native, no extra container. Same pooler URL. Cleaner but requires a config-rewrite + make configure redeploy.
  3. App-side spans — the polymarket-fetch services already emit sqlx queries as spans via tracing-opentelemetry. SigNoz Services → DB Calls panel surfaces per-query timing, error rates, and rate-of-calls. This is what we have today.

Current decision: stick with #3 (app-side spans). The DB-call surface in SigNoz already covers slow-query identification (the bot’s WITH targets AS (...) query showing up as a 1.6s span every 2min is what flagged it), and per-table query rate is observable through trace attributes. Adding (1) or (2) is +ev only if we start needing connection-pool saturation alerts, transaction-rate dashboards, or per-table I/O — none of which the current workload demands.

Re-evaluate if any of:

  • Bot/scheduled-run start hitting too many connections errors (current Supabase pooler limit is 60 for session mode).
  • Query latency p99 climbs past 5s without an obvious code change.
  • We need to alert on table bloat or index health.

Operational checklist when adding a new app

  1. Drop the observability.rs snippet (or equivalent SDK init) in the app crate; call init() from main.
  2. Deploy via the apps role (manifest entry with default telemetry enabled — no extra config needed).
  3. Wait ~30s for the first batch.
  4. Verify: services view in SigNoz lists the new OTEL_SERVICE_NAME value with non-zero callRate.
  5. Logs view: filter service.name=<name> — first batched logs appear within ~30s of first info!()/warn!()/error!().

If signals don’t show up:

SymptomLikely causeFix
Service not in services listOTEL_EXPORTER_OTLP_ENDPOINT not reaching collectordocker exec apps-<name> env | grep OTEL should show host.docker.internal:4317; if missing, the apps role didn’t render the env block (check item.telemetry not set to false).
Service in list but call rate zeroApp not using #[tracing::instrument] or equivalent spansWithout spans, only the implicit “root span per request” emits. Wrap your work in tracing::span! / #[instrument].
otelcol_exporter_send_failed_* non-zeroSigNoz ingestion key invalid, network blocked, or rate-limitedCheck secrets/signoz_ingestion_key; check otelcol-contrib journal journalctl -u otelcol-contrib -n 50.
Logs missing but traces presentopentelemetry-appender-tracing bridge not initializedConfirm OpenTelemetryTracingBridge::new(&logger_provider) is one of the layers passed to tracing_subscriber::registry().

Graph View

Backlinks