Guided runbook for the local OpenTelemetry store on pmv2-zurich that replaced the expired SigNoz Cloud workspace. Covers the architecture, the otelq query wrapper, and the emergency procedures for the things most likely to break.
Coverage gap (pre-existing, not caused by this switch)
pmv2-position-manager (PM), pmv2-shortterm-crypto-* (emit/observe), and pmv2-telegram-connector (Rust services) do NOT export logs to the collector. They emit traces + metrics fine, but their tracing-opentelemetry crate has no log-exporter by design — see the coverage gap section below. For those services, docker logs <container> is still how you read logs.
Why the local store exists
The SigNoz Cloud workspace we had been shipping to (ingest.eu2.signoz.cloud:443) expired. The collector’s journal was flooded with:
workspace does not exist
Invalid or missing key
Rather than renew SigNoz Cloud or migrate to another vendor mid-flight, we cut the exporter over to three local file exporters writing JSONL to /var/log/otel/ on the VM itself, and installed duckdb + a helper otelq wrapper for querying.
Trade-offs vs SigNoz Cloud:
No web UI, no dashboards, no cross-service trace waterfall view.
Retention is disk-bounded (rotation defaults: 7d logs / 3d traces / 1d metrics, capped backups). Metrics survive the shortest.
Query surface is DuckDB SQL over JSONL, wrapped by otelq for the common cases.
Files are 0644 (world-readable) so no sudo is needed to query.
Zero external dependency — nothing to expire, no ingestion key to rotate. Works offline.
To bring back SigNoz Cloud (or any OTLP vendor) later, see emergency step 6 — both file and OTLP exporters can run concurrently on the same collector.
Files are 0644 (world-readable) via systemd drop-in Umask=0022 at /etc/systemd/system/otelcol-contrib.service.d/umask.conf; ops is a member of the otelcol-contrib group — no sudo required for querying
Query engine
DuckDB v1.1.3 at /usr/local/bin/duckdb
Query wrapper
otelq at /usr/local/bin/otelq — bakes in the OTLP-JSON unnest CTE
Config source of truth
ansible/roles/monitoring/templates/config.yaml.j2 in /Users/levander/levandor/terraform (never hand-edit /etc/otelcol-contrib/config.yaml on the VM — re-apply the role)
otelq cheat sheet
All commands run as ops on pmv2-zurich. No sudo needed.
otelq errors [hours=1] [limit=50] recent ERROR+FATAL (severityNumber >= 17)
otelq recent [minutes=15] [limit=100] all recent lines
otelq svc <service> [minutes=30] [limit=100] filter by service.name
otelq grep <pattern> [minutes=30] [limit=100] regex over body
otelq stats [hours=1] counts by service + severity
otelq services [hours=1] log volume per service
otelq sql "<query>" arbitrary DuckDB query (uses the FLAT_LOGS CTE)
otelq raw interactive DuckDB REPL (no baked-in CTE)
otelq disk du of /var/log/otel
For anything ad-hoc, drop into SQL. otelq sql provides a FLAT_LOGS CTE that already unnests resourceLogs[].scopeLogs[].logRecords[].observedTimeUnixNano and KVList attributes so service_name, body, severity_text, severity_number, ts are top-level columns.
$ otelq sql "SELECT service_name, count(*) AS n
FROM FLAT_LOGS
WHERE ts > now() - INTERVAL 1 HOUR
GROUP BY 1 ORDER BY n DESC"
Coverage gap — Rust services without a log bridge
docker logs remains the log path for these services
Root cause: their tracing-opentelemetry crate ships spans + metrics only, and does not include a log exporter by design. Fix requires adding the opentelemetry-appender-tracing bridge in each service (same pattern the polymarket-fetch services and cohort/weather algos use — see the code snippet in [[observability-flow#srcobservabilityrs|src/observability.rs]]). This is not owned by the deploy layer; it needs a code change in each service repo.
Cross-refs: the solana agent’s finding on this is captured in the shortterm-crypto vault notes / babylon dm:deploy+positionmanager #1042. Traces + metrics for these services are in /var/log/otel/traces.jsonl and metrics.jsonl respectively — only logs are absent.
Roll through these in order when something looks off. Each step is self-contained and idempotent unless flagged otherwise.
1. otelq: command not found or DuckDB missing
Symptom
Running otelq … gives command not found, or otelq errors out complaining about duckdb.
Cause: the monitoring role installs both otelq and duckdb. Fresh VMs, image swaps, or manual removals can leave one or both missing. make deploy does NOT cover this — that target runs the apps role only.
Fix (from the terraform repo, idempotent):
cd /Users/levander/levandor/terraformansible-playbook ansible/site.yml --limit pmv2-zurich --tags monitoring
This re-installs otelcol-contrib, DuckDB, and re-templates /usr/local/bin/otelq from the role.
2. /var/log/otel/ empty or files aren’t growing
Symptom
Directory is empty, or logs.jsonl mtime is stale, or otelq recent returns zero rows.
The collector is either dead or failing to write. Check on the VM:
That drops the numbered backups (logs.jsonl.1, logs.jsonl.2.gz, etc.) but keeps the three active files (no suffix). The collector keeps writing to those without restart.
For a permanent change, edit ansible/roles/monitoring/defaults/main.yml in the terraform repo (adjust otel_*_retention_days and/or the backup caps) and re-apply the role.
4. Query returns Permission denied
Symptom
otelq recent fails with a filesystem Permission denied error, or DuckDB can’t open /var/log/otel/logs.jsonl.
Two likely causes:
You’re not on pmv2-zurich. SSH in first. The /var/log/otel/ files live on the VM, not on your workstation.
The ops user isn’t in the otelcol-contrib group anymore. Rare — happens if someone rebuilt the user manually.
Then log out and back in (group membership refreshes on new session).
5. Query returns Binder Error: Could not find key…
Symptom
otelq recent or any log query returns a DuckDB Binder Error: Could not find key "..." referencing something like observedTimeUnixNano or an attribute path.
OTel changed its record shape (upstream schema evolution). The current FLAT_LOGS CTE in /usr/local/bin/otelq assumes:
resourceLogs[].scopeLogs[].logRecords[].observedTimeUnixNano for timestamps
KVList-shaped attributes[] for resource + record attributes
severityNumber (int) and severityText (string) on each logRecord
Fix: update the FLAT_LOGS CTE in /usr/local/bin/otelq. Source of truth is the “Install otelq” task in ansible/roles/monitoring/tasks/main.yml in the terraform repo. Update there, then re-apply the role (step 1). Do NOT patch /usr/local/bin/otelq directly on the VM — it’ll get overwritten on the next make configure.
6. Bring SigNoz Cloud (or an equivalent OTLP vendor) back
File exporters and OTLP exporters can run in parallel
The collector supports fan-out. Adding an OTLP exporter alongside the existing file exporters means every signal is written locally and shipped to the vendor — good for a migration window where you want overlap before turning file exporters off.
Steps:
Get a fresh SigNoz Cloud ingestion key (or the equivalent OTLP key + endpoint from the new vendor).
Stash it locally: secrets/signoz_ingestion_key (mode 0600, gitignored).
Edit ansible/roles/monitoring/templates/config.yaml.j2 in the terraform repo. Add the OTLP exporter back:
And attach it to each pipeline alongside the file exporter — e.g. exporters: [file/logs, otlp_grpc].
Re-apply the role:
cd /Users/levander/levandor/terraformansible-playbook ansible/site.yml --limit pmv2-zurich --tags monitoring
Confirm signals are flowing to both destinations. Local: otelq recent 5 shows lines. Vendor: use the vendor’s UI. Collector self-metrics: curl -s http://127.0.0.1:8888/metrics | grep -E 'otelcol_exporter_(sent|send_failed)' should show sent climbing on both exporters and no send_failed.
Once the vendor is proven stable and you don’t need local retention anymore, drop the file exporter branch from each pipeline and re-apply. Or keep both indefinitely — the file store is cheap disk.
7. Config source of truth
Do NOT hand-edit config on the VM
Every knob lives in git under ansible/roles/monitoring/ in the terraform repo. The rendered config on the VM is /etc/otelcol-contrib/config.yaml, but it’s regenerated on every make configure / role re-apply. Hand-edits will disappear the next time anyone runs the role.
Files that own the pipeline:
Path
Owns
ansible/roles/monitoring/templates/config.yaml.j2
The full otelcol-contrib config (receivers, processors, exporters, pipelines)
PM’s sqlx pool may not amortize the Zurich↔Dublin RTT — the gates_ms telemetry shows medians ~730 ms for the four sequential config SELECTs pre-book-fetch. Investigate with otelq svc pmv2-position-manager 30 for spans (traces DO ship from PM — the coverage gap is logs-only, see the coverage gap section).