Levandor

mando-cli — always-on OTel trace capture + `mando tail`

3 min read

Two features that landed together in the mando-cli working tree 2026-07-28 (UNCOMMITTED, pending user smoke + push): (1) every mando up stack now captures OTel traces to disk and can fan out to Datadog APM, and (2) a new local-only mando tail with a zero-query-language substring filter over gron-flattened telemetry.

For Agents

Both features are LOCAL-first. Trace capture writes raw OTLP JSON to a workspace file; tail reads local capture files only. For remote telemetry use Datadog (follow the dd_query/dd_url pointers from mando-cli-flow-run-2026-07-28). Architecture consolidation left exactly ONE piped-child streaming seam and ONE OTLP walk/reader — see Consolidation below.

Feature 1 — always-on OTel trace capture + Datadog APM

Every mando up stack now includes a mando-otel-collector (contrib 0.109.0).

  • OTLP receivers on 4317 (gRPC) / 4318 (HTTP); host ports overridable via MANDO_OTEL_GRPC_PORT / MANDO_OTEL_HTTP_PORT.
  • Writes raw OTLP JSON as NDJSON to <workspace>/.otel/traces/spans.ndjson (rotation 50 MB / 3 backups).
  • Services get OTEL_EXPORTER_OTLP_ENDPOINT injected only when absent — docker network: mando-otel-collector:4318; host: localhost honoring the port var.

mando up --datadog now ALSO ships APM: the Datadog agent gains an OTLP receiver and the collector fans out via otlphttp to the agent (in addition to the on-disk capture).

ensure_infra config-revert bug (review catch, fixed)

ensure_infra runs on EVERY command and originally hardcoded the base collector config on disk. So any mando status after up --datadog reverted the collector config, silently killing APM forwarding on the next collector restart. Fixed by deriving the collector variant from the presence of the datadog overlay file — state lives in the file, so all writers agree.

Feature 1b — mando flow run bundle/span flags

  • --span-file <path> — extracts THIS run’s raw spans, matched by the flow.exec_id span attribute (± 30s grace), scanning rotated capture files, written verbatim (lines unchanged).
  • --bundle <dir> — writes logs.ndjson + spans.ndjson + params.json, where params.json is the exact POST body. Any bundle is therefore re-runnable via --params bundle/params.json.

Feature 2 — mando tail

mando tail [logs|traces|all] [FILTER ...] [--svc <alias>] [--since 10m] [-f] [--flat]

Zero query language. Each line is gron-flattened into dotted assignments:

  • nested objects → dotted paths (flow.exec_id = "...")
  • arrays → [i] indices
  • non-JSON lines → raw = "..."
  • scalars → value = ...

Every FILTER term must substring-match some assignment (terms AND together). Because mando uses dd-style root keys, the natural filter reads flow.exec_id = "<id>.

  • Default output is verbatim (golden-file friendly).
  • --flat prints only the MATCHING assignments (exploration mode).
  • Local-only by design — remote telemetry is Datadog; use the dd_query pointers.

Consolidation win

  • Exactly ONE piped-child streaming seam: system/logstream.rs, shared by flow run and tail.
  • Exactly ONE OTLP walk + capture-file reader: flow/spans.rs.

Process & gate

Recovery note

The tail-fix subagent died mid-edit on a weekly usage limit, leaving a half-applied signature change (tree didn’t compile). The orchestrator completed the fixes inline (including deleting the dead fixer remnant); both reviewers re-verified CLOSED.

  • Gate: 532 lib + 8 integration + smoke, 0 failures; clippy 0; release 0.

Graph View

Backlinks