Levandor

Rust OTel Logs Not Reaching SigNoz — Root Cause & Fix

4 min read

The “house-wide Rust OTel log-export bug” — services show traces and metrics in SigNoz but never logs — is not a bug. tracing-opentelemetry does not export logs by design; the fix is to wire the separate opentelemetry-appender-tracing bridge as its own pipeline.

For Agents

Cross-project finding. Affects every Rust service on deploy’s fleet that ships OTLP to SigNoz Cloud — confirmed symptom on pmv2-position-manager, cohort, tg. Root-caused during the slv1 measurement-rig spec-grounding pass (2026-07-03) and relayed to the deploy agent via babylon DM #1040. If you are wiring OTel logs in any Rust service, use the recipe below.

Symptom

  • Service appears in SigNoz for traces and metrics, but no logs ever show up under that service.
  • No error at startup; the OTLP exporter connects fine (traces/metrics prove the pipeline and auth work).
  • Reproduces on every fleet service using the house tracing-opentelemetry + opentelemetry-otlp (gRPC) setup.

Root Cause

tracing-opentelemetry does not export logs at all — by design. From docs.rs, verbatim:

Logging to OpenTelemetry collectors is not supported by this crate, only traces and metrics… consider opentelemetry-appender-tracing.

The fleet wired tracing-opentelemetry for the trace/metric bridge and assumed logs rode the same pipeline. They never do — there is no layer translating tracing events into OTel log records, so nothing is emitted on the logs signal. This reproduces the symptom exactly (no service in SigNoz logs; traces + metrics fine).

It is a wiring gap, not upstream immaturity

The OpenTelemetry-Rust logs signal itself is Stable as of mid-2026. Nothing is blocked — the third pipeline was simply never wired.

Fix — Known-Good Recipe

Wire a separate logs pipeline alongside the existing trace/metric one:

  1. opentelemetry-appender-tracing → provides OpenTelemetryTracingBridge, a tracing_subscriber layer that converts tracing events into OTel log records.
  2. OTLP LogExporter (same endpoint / collector as traces + metrics).
  3. SdkLoggerProvider + BatchLogProcessor (batches log records for export).
  4. Add OpenTelemetryTracingBridge::new(&logger_provider) as a third tracing_subscriber layer — alongside the fmt layer and the tracing-opentelemetry trace layer.

Footguns

  • Feedback loop: the OTLP exporter uses tonic / opentelemetry internally, which themselves emit tracing events. If those reach the bridge you get a log → export → log → export loop. Filter the exporter’s own tonic / opentelemetry targets out of the bridge layer.
  • Lost batches on shutdown: call logger_provider.shutdown() on SIGTERM (flush) or the last BatchLogProcessor batch is dropped silently. Fold it into the same graceful-shutdown flush that already handles the tracer / meter providers.
  • EnvFilter eating events: make sure the EnvFilter (or per-layer filter) does not filter the events out before they reach the bridge layer.

Version Context & Pinning

  • The OTel stack (opentelemetry / opentelemetry_sdk / opentelemetry-otlp) on the fleet and the slv1 rig is 0.31.x — traces + metrics.
  • Match tracing-opentelemetry to the OTel stack — don’t just take the newest. tracing-opentelemetry 0.33 requires opentelemetry 0.32; to stay on the 0.31 OTel stack you must pin tracing-opentelemetry 0.32. (The slv1 rig plan originally specced tracing-opentelemetry 0.33 + otel 0.31 — that pair does not resolve; the build pinned tracing-opentelemetry 0.32 to keep otel at 0.31.) Bumping tracing-opentelemetry to 0.33 drags the whole OTel stack to 0.32.
  • Logs require the separate opentelemetry-appender-tracing crate — not pulled in transitively, and it too must be version-matched to the OTel stack.

Fleet Impact

  • Confirmed-affected deploy fleet services: pmv2-position-manager, cohort, tg.
  • Their load-bearing log path is currently journald + Postgres stats, not SigNoz — this finding explains why, and unblocks wiring OTel logs whenever wanted.
  • slv1’s solv1-rig keeps journald + Postgres stats as its Week-1 log path; the bridge is a wired-when-wanted add-on (spec §7).

Graph View

Backlinks