Voice Agent Activity Log

For Agents

Reverse-chronological session log. Newest entries at top, grouped by date (## YYYY-MM-DD). Each bullet: one piece of work, short summary, wikilinks to docs touched. Updated by obsidian-documenter on every project doc write. Read by historian at bootstrap.

2026-07-21

• Project initialized — multi-tenant AI receptionist SaaS, Rust-native cascaded voice pipeline on Twilio Media Streams. Repo ~/coding/voice_agent (greenfield). Index + 8 research notes created — voice-agent
• HEADLINE FINDING — ~480ms of a measured 1,048ms Pipecat+Twilio voice-to-voice loop is unaccounted orchestration overhead, not vendor service time (STT TTFB 195ms + Smart Turn/VAD 84ms + LLM TTFT 268ms + TTS TTFB 23ms = ~570ms of actual service; residual = base64 framing, 20ms packetization, jitter buffering, VAD windowing). This residual is the entire justification for writing the pipeline in Rust. Production fleet reality 680ms p50 / 1,180ms p95, real SLA target <1,400ms p95; every platform roughly triples p50→p95 so optimize p95 (where callers hang up). Perception ladder: ~200ms human turn-gap median (Stivers et al., primary source UNVERIFIED) · ~300ms noticed · ~500ms rhythm breaks · ~800ms unnatural — note the measured loop is already past “unnatural”. TTS connection warmth is worth ~250ms: same Aura-2 model measured at 313ms P50 cold (Coval) vs 23ms warm (fullstackml) — pre-warm and pool TTS WebSockets — voice-agent-latency-budget-2026-07-21
• TTS: the binding constraint is CONCURRENCY, not price or quality — ElevenLabs caps at 15 concurrent even on the $990/moBusinesstier,identicaltothe$299 tier (Pro $99=10);CartesiaScalealso15;RimeStarter20;\ast \ast Deepgram45PAYG/60Growth,nosalescall\ast \ast .This\ast \ast disqualifiesElevenLabsandCartesiafromself-servemulti-tenantinboundtelephonyregardlessofquality\ast \ast .DoNOTpickonarenarank—ArtificialAnalysisSpeechArena(76models,blind,June2026)top-4spans\ast \ast 24ELOwith\pm 16CIs=statisticallynoise\ast \ast ;pickonlatency\ast \ast IQR\ast \ast ,concurrency,\mu -lawsupport.IQR>P50forphones:Cartesia188msP50/100msIQR\ast feelsworse\ast thanElevenLabs288msP50/28msIQR.Pricing/min-of-speech:Inworld$0.009 · Aura-2 $0.024–0.027\cdot Cartesia$0.034 · EL Flash v2.5 $0.045\cdot ELMultilingual$0.090. ElevenLabs v3 is batch-only, NOT realtime (docs point realtime users at Flash v2.5; Turbo v2.5 deprecated) — so its expressive tags [hesitates]/[stammers]/[pauses]/[laughs]/[sighs], exactly what a receptionist wants, are structurally unavailable. Rime is the only vendor explicitly engineering spontaneous speech (Arcana: fillers/pauses/prosodic stress; Mist v2: backchanneling + breathing; trained on conversational not scripted speech) — BLOCKER: μ-law 8k unverified. PlayHT/Play.ai is dead (Meta acqui-hire, API shut 2025-12-31) — its presence in any comparison = staleness signal. SKEPTICISM: Coval numbers reach us via Gradium’s blog and Gradium tops that benchmark; raw dashboard JS-rendered, not extractable — voice-agent-tts-vendor-landscape-2026-07-21
• STT: Deepgram Flux is the pick (flux-general-en, wss://api.deepgram.com/v2/listen — note /v2, not /v1) — fuses end-of-turn detection into the acoustic model (~260ms EOT p50, but p90 1s / p95 1.5s); native encoding=mulaw&sample_rate=8000 so Twilio frames go in with zero transcoding; EagerEndOfTurn/TurnResumed pair fires 150–250ms before confirmed EOT enabling speculative LLM dispatch with cancel (costs 50–70% extra LLM calls — trivial at Haiku prices). Tuning: eot_threshold 0.5–0.9 (def 0.7), eager_eot_threshold 0.3–0.9, eot_timeout_ms 500–60000. Deepgram has the only first-party Rust SDK in the entire field (crate v0.10.0, 2026-07-12) + a Twilio↔Rust tutorial — VERIFY it covers /v2/listen not just /v1. AssemblyAI rejected: cheaper ($0.0025/min, 307ms P50 word emission) but max_turn_silence defaults to 1280ms ≈ 5x slower than Flux to commit a turn, and bills by SESSION duration not audio duration. CRITICAL CAVEAT: Artificial Analysis’ streaming STT index contains NO narrowband telephony audio at all (50% AA-AgentTalk / 25% VoxPopuli parliamentary / 25% Earnings22 — all wideband); every published leaderboard position is measured on audio a G.711 trunk will never deliver, and nobody publishes per-domain telephony WER → budget a week to A/B on recorded 8kHz μ-law from the actual trunk with Nova-2 phonecall as control (only model explicitly special-cased for narrowband) — voice-agent-stt-vendor-landscape-2026-07-21
• Speech-to-speech DISQUALIFIED for a receptionist — three documented failure modes, all first-turn — (1) Alphanumeric read-back corruption: the transcript is correct, the audio-generation path corrupts the value. Booking ref MAR17M → MAR178 → MAR179 → MAR17F → tool call fired with the wrong value; developer measured ~70% accuracy; same prompt on cascades (ElevenLabs conversational, Vapi+GPT-4o) had no issues. Sierra τ-voice (arXiv 2603.13686) confirms: “Authentication is the dominant bottleneck… agents fail to transcribe names and emails even when spelled letter-by-letter.” (2) Language/accent drift triggered by caller NAMES — gpt-realtime-2.1, 2026-07-21: “If I call Anastasia, it changed to Russian. Amir, arabic.”; identical bug on Gemini Live over Twilio (Spanish on “Jose”) — for inbound, the first thing a caller says is their name. (3) No verbatim guarantee and the transcript is not what the caller heard — hallucinated monologues in audio appearing in NO transcript log; OpenAI’s own guide admits the model “may prioritize the exact wording of an instruction over the broader behavior you intended”; xAI shipped force_message precisely because prompting doesn’t work — explicitly not in the OpenAI Realtime API. Secondary: tool calls decided from audio not clean text (LiveKit); interruption selectivity worst-in-class — OpenAI 6% (τ-voice); 8kHz is out-of-distribution for models trained on 16–48kHz. COUNTER-EVIDENCE recorded: Sierra ships GPT-Realtime-2 at ~30% faster P50 / up to 200% faster P90 vs their own cascade — but only by building “an agent harness that provides the control plane” (supervisor, PII redaction, tracing, and custom-tuned VAD replacing the model’s own turn detection) — they rebuilt a cascade’s control plane around the S2S core. Cost no longer disqualifies S2S generally (Gemini Live ~$0.023/min\approx cascadeparity)—onlyOpenA{I}^{\prime }s($0.073–0.123/min, scales with conversation length since full context re-sends each turn); but Gemini Live is PCM16 16k/24k only (no μ-law) with a 15-min audio session limit / ~10-min WebSocket lifetime — voice-agent-speech-to-speech-disqualification-2026-07-21
• Telephony: Twilio wins despite being the most expensive — Media Streams constraints: mulaw 8k only, ONE bidirectional stream per call, DTMF inbound-only on bidi; $0.0129/minall-in(inbound$0.0085 + streaming $0.0044)+$1.15/mo per number. Cheaper: Plivo $0.006(streamingfree),SignalWire$0.0066, Telnyx $0.0097—\ast \ast atSMBvolumethespreadis$6 vs $14/month,i.e.noise\ast \ast .Telnyxtrap:headline\ast \ast$0.0032 is not what you pay** (Call Control +$0.002additive,mediastreaming+$0.0035). The real differentiator is account-suspension risk when you carry other businesses’ numbers: Telnyx HIGH (FCC proposed ~$4.5Mfine\ast \ast forKYCviolations;accountssuspendedwithinminutesover\ast \ast$0.06 negative balances) · Plivo MEDIUM (ticket-only support) · SignalWire MEDIUM-HIGH for small accounts · Twilio LOWEST — this alone justifies the most expensive carrier. Carrier RTT (published on Telnyx’s own site, discount accordingly): Telnyx 71/118ms, Twilio 89/161ms, Vonage 94/152ms — the 18ms p50 delta is not worth switching; co-locating in us-east-1 near Twilio’s us1 media servers matters more than carrier choice. Twilio ConversationRelay: AVOID — $0.07/min on top of voice minutes (~5.5x DIY) and it’s a text-only WebSocket with no raw audio in either direction, so custom VAD, endpointing, barge-in and DSP are all impossible by construction — voice-agent-telephony-carriers-2026-07-21
• Managed platforms fail on CAPABILITY before price — NO managed platform (Vapi, Retell, Bland, ElevenLabs, Telnyx) lets you upload or host your own VAD/turn-detection model; only LiveKit Agents, Pipecat, or your own pipeline qualify. onNumberSeconds (separate endpointing hold time for digit strings) exists ONLY on Vapi and Telnyx — for a receptionist taking phone numbers and addresses this is the single highest-value knob, and Retell/Bland/ElevenLabs cannot do it, so callers reading back a phone number get cut off mid-digit. Retell exposes only opaque 0–1 dials (responsiveness, interruption_sensitivity) with no millisecond visibility — the same root cause as its most-cited criticism, weak observability. Bland’s entire turn-taking surface is three fields; ElevenLabs’ is a 3-value enum + timeout. Vapi’s waitFunction (user-supplied math expression mapping end-of-utterance probability → wait ms, default "200 + 8000 * x") is the most powerful knob on any managed platform. Reliability: Vapi status page shows ~19+ hours of call-affecting degradation in a 10-week window including a 4h07m critical outage from database connection exhaustion (2026-05-21), and a weekly release channel that explains “working agents break when the platform updates”. LiveKit Agents GitHub issues are almost entirely interruption bugs — #3702 (tool-call results lost on interruption → duplicate order creation in production) and #3685 (“Latency doubles in telephony contexts”). Pipecat: 3–8s cold start (sequential service init) and aggregation_timeout=1.0 adds an unavoidable 1s delay to EVERY response — voice-agent-managed-platform-control-surfaces-2026-07-21
• Cost model — the LLM is ~10% of cost, not the driver — recommended build for a 3-min/12-turn call (agent speaks ~1.1 min ≈ 1000 chars TTS): Twilio $0.0129+DeepgramFlux$0.0065 + Aura-2 $0.0100+\ast \ast Haiku4.5$0.0032** = $0.0326/min=$0.098 per 3-min call. Haiku 4.5 with prompt caching cuts LLM cost 65% to $0.0096/call;\ast \ast TTScosts3–5xtheLLM—optimizeTTS,notthemodel\ast \ast .Corollary:speculativeLLMdispatch(+50–70$0.085 (+161%) · Retell $0.073–0.088\cdot ElevenLabsAgents$0.10–0.13 (+207–299%) · Bland $0.11–0.14\cdot LiveKitCloud$0.045 · Pipecat Cloud ~$0.04–0.058.\ast \ast Build-vs-buybreak-evenvsVapi:600,000minutes\approx 200,000three-minutecalls\ast \ast at8engineer-weeks($30k) — i.e. cost is the tiebreaker, not the argument; the build case rests on capability (the 480ms residual and custom VAD are unreachable on any managed platform) — voice-agent-cost-model-2026-07-21
• Rust ecosystem: there is no framework to adopt — no voice-agent framework has a Rust agent runtime. LiveKit’s own README lists Agents SDKs as Python and Node.js only; Rust appears solely under client SDKs marked “Developer Preview, not production-ready”. Pipecat is Python-only. flowcat (https://github.com/AreevAI/flowcat, Apache-2.0) is a native-Rust Pipecat-architecture runtime with native SIP/RTP, 9 telephony serializers, 20+ STT / 29+ TTS / 23+ LLM providers — architecturally ideal but 74 stars, v0.1.0, only the Gemini Live + Plivo path verified end-to-end (neither this project’s architecture nor its carrier), and its “p99 0.61ms” benchmark measures frame-routing overhead, not voice-to-voice latency → watch, don’t bet. Where Rust is first-class: the Deepgram crate, plain Twilio WS+JSON, and OpenAI Realtime over SIP (plain WSS/WebRTC/SIP). The gap is entirely at the orchestration layer — exactly the layer this project intends to own — voice-agent-rust-ecosystem-status-2026-07-21
• Open items to verify before spending — Rime μ-law 8k support · that deepgram crate v0.10.0 covers /v2/listen (Flux) not just /v1 · Deepgram’s Streaming-vs-PreRecorded pricing column order (two fetches disagreed) · the raw Coval dashboard (benchmarks.coval.ai/tts, JS-rendered) in a real browser · Hume Octave 2 pricing (sources contradict 3.3x) · and an own A/B on recorded 8kHz μ-law from the actual Twilio trunk (the highest-value item — the only one that could change the vendor decision) — voice-agent