Where the milliseconds actually go in a cascaded voice pipeline on Twilio, and why roughly half the voice-to-voice latency is not attributable to any vendor. This is the single finding that justifies writing the pipeline in Rust instead of adopting Pipecat or LiveKit Agents.
The Headline Measurement
~480ms of a 1,048ms loop is unaccounted orchestration overhead
A measured Pipecat + Twilio voice-to-voice loop totalled 1,048ms. Summing the four service components accounts for only ~570ms of it.
Component Latency Notes STT TTFB 195ms Smart Turn / VAD 84ms LLM TTFT 268ms TTS TTFB 23ms Warm pooled connection — see below Sum of services ~570ms Measured total 1,048ms Unaccounted residual ~480ms base64 framing, 20ms packetization, jitter buffering, VAD windowing [measured] — https://www.fullstackml.dev/p/15-where-does-the-time-go-measuring
The residual is not mystery latency. It is the sum of the things a framework does between the services: base64-encoding every audio frame for the Twilio WebSocket JSON envelope, 20ms packetization boundaries, jitter buffering, and VAD analysis windows. In a Python framework these costs are structural — they live in the event loop, the serialization layer, and the queue hops. A Rust pipeline can attack all four directly.
For Agents
The design thesis of this project in one line: the vendors are already fast enough (~570ms of real service time); the framework is what makes it feel slow. Every architecture decision in
~/coding/voice_agentshould be evaluated against “does this reclaim part of the 480ms residual?”
Production Reality vs. Lab Numbers
The 1,048ms figure is a clean single-call measurement. Fleet-level numbers are worse and have long tails.
| Metric | Value | Source |
|---|---|---|
| Production p50 | 680ms | destilabs 2026 benchmark [measured] |
| Production p95 | 1,180ms | same |
| Recommended real SLA | < 1,400ms p95 | same |
https://www.destilabs.com/blog/ai-voice-agent-benchmark-2026
Every platform roughly triples p50 → p95
Optimize p95, not p50. p50 is what demos measure; p95 is where callers hang up. A pipeline with an excellent p50 and an uncontrolled tail will read as broken to a meaningful fraction of callers.
Perception Thresholds
What the latency number actually means to a human on the other end of a phone line.
| Threshold | Effect |
|---|---|
| ~200ms | Median turn gap in human conversation — the target if you want it to feel natural |
| ~300ms | Users start to notice a delay |
| ~500ms | Conversational rhythm breaks |
| ~800ms | Feels distinctly unnatural |
Citation caveat
The ~200ms human-conversation median turn gap is universally attributed to Stivers et al., but the primary source was [unverified] at research time. Treat the exact number as directional. The ordering of the thresholds is not in dispute; the precise 200ms is.
Note the uncomfortable implication: the measured 1,048ms loop sits well past the ~800ms “unnatural” line, and the production p95 of 1,180ms is worse. Nobody in this market is currently delivering natural-feeling turn-taking. Getting under ~800ms p95 would be a genuine differentiator, and the 480ms residual is where that budget has to come from.
TTS Connection Warmth Is Worth ~250ms
Same model, 313ms vs 23ms, purely from connection reuse
Two independent measurements of Deepgram Aura-2 TTFB:
Source TTFB Connection model Coval 313ms P50 Cold per-request connection fullstackml 23ms Persistent pooled WebSocket The ~290ms delta is TLS handshake + WebSocket upgrade + cold-path warmup, paid on every single utterance if you don’t pool.
Implication for the build: pre-warm and pool TTS WebSocket connections per tenant, and keep them alive across turns within a call. This is one of the cheapest large wins available and it is invisible in vendor benchmarks — vendors publish numbers that assume you did it, and naive integrations don’t.
Skepticism flag on the Coval figures
The Coval TTS numbers reach us via Gradium’s blog post, and Gradium tops that benchmark. The raw dashboard at
benchmarks.coval.ai/ttsis JS-rendered and was not extractable at research time. The 313ms cold-connection figure is directionally consistent with a TLS+WS handshake cost, which is why it’s usable here — but it is second-hand from an interested party. See open items in voice-agent.
Budget Allocation for the Rust Build
Working backwards from a < 800ms p95 target, using Deepgram Flux + Haiku 4.5 + pooled Aura-2:
| Stage | Budget | Lever |
|---|---|---|
| Carrier RTT | ~90ms p50 / ~160ms p95 | Co-locate in us-east-1 near Twilio us1 media servers |
| STT + EOT detection | ~260ms p50 (Flux EOT) | eot_threshold tuning; EagerEndOfTurn speculative dispatch |
| LLM TTFT | ~270ms | Prompt caching; speculative dispatch hides most of this |
| TTS TTFB | ~25ms | Requires warm pooled WS — 313ms if you skip it |
| Orchestration | target ≪ 480ms | The Rust thesis |
Flux
EagerEndOfTurncollapses the LLM termFlux fires an
EagerEndOfTurn/TurnResumedevent pair 150–250ms before confirmed end-of-turn. Dispatching the LLM speculatively on the eager event and cancelling onTurnResumedhides most of the 270ms TTFT behind the endpointing window. Costs 50–70% extra LLM calls — trivial at Haiku prices, see voice-agent-cost-model-2026-07-21. Details in voice-agent-stt-vendor-landscape-2026-07-21.
Sources
- https://www.fullstackml.dev/p/15-where-does-the-time-go-measuring — component attribution of the 1,048ms loop [measured]
- https://www.destilabs.com/blog/ai-voice-agent-benchmark-2026 — production fleet p50/p95 [measured]
- Coval TTS benchmark, via Gradium blog — Aura-2 cold TTFB [measured, but second-hand from an interested party]
- Stivers et al. — human turn-gap median [unverified primary source]