Speech-to-speech (OpenAI Realtime, Gemini Live) is rejected for this product. Three documented, reproducible failure modes are each individually disqualifying for a receptionist, and all three land on the first turn of a call. This note records the evidence, including the counter-evidence.
The decision in one line
A receptionist’s core job is capturing names, phone numbers, emails, and booking references correctly. S2S models are documented to corrupt exactly those, switch language when they hear a foreign-sounding name, and emit audio that does not appear in the transcript. The failure modes are aimed directly at the use case.
Failure Mode 1 — Alphanumeric Read-Back Corruption
The transcript is correct; the audio-generation path corrupts the value
This is the subtle and dangerous part. Logging looks clean. The caller hears something different.
Documented case: booking reference MAR17M, read back as:
MAR178MAR179MAR17F- …then a tool call fired with the wrong value
The developer measured roughly 70% accuracy on alphanumeric read-back. The same prompt on cascaded pipelines (ElevenLabs conversational, Vapi + GPT-4o) had no issues — this is not a prompting problem, it’s an architecture property.
Independent confirmation from Sierra’s τ-voice benchmark:
“Authentication is the dominant bottleneck… agents fail to transcribe names and emails even when spelled letter-by-letter.”
arXiv 2603.13686
The “even when spelled letter-by-letter” clause is the killer. The standard mitigation for a mishearing — asking the caller to spell it — does not work.
Failure Mode 2 — Language / Accent Drift Triggered by Caller Names
Reported against
gpt-realtime-2.1on 2026-07-21“If I call Anastasia, it changed to Russian. Amir, arabic.”
The identical bug is reported on Gemini Live over Twilio: switches to Spanish on hearing “Jose.”
The model treats a name’s phonetic origin as a language cue and switches output language mid-call.
Why this is fatal specifically for an inbound receptionist: the first thing a caller says is very often their name. This is not an edge case that shows up in the long tail of a conversation — it is a first-turn failure, on the most common possible opening utterance, and it is worse for exactly the multicultural customer base a business would most want to serve well.
Failure Mode 3 — No Verbatim Guarantee, and the Transcript Is Not What the Caller Heard
Three separate pieces of evidence, all pointing the same direction:
- Documented hallucinated monologues in the audio that appear in NO transcript log. The audio channel and the text channel are not the same artifact. Your observability is lying to you by construction.
- OpenAI’s own prompting guide admits it. The model “may prioritize the exact wording of an instruction over the broader behavior you intended.” There is no supported way to say “read this string back exactly.”
- xAI shipped a
force_messageprimitive precisely because prompting doesn’t work. A vendor building a competing S2S product concluded the problem was unsolvable at the prompt layer and added an API primitive for it. That primitive is explicitly not part of the OpenAI Realtime API.
For Agents
Point 3 is the strongest single piece of evidence in this note. It’s not a user complaint or a benchmark — it’s a competing vendor’s product decision that concedes the failure mode exists and cannot be prompted away.
For a receptionist that must confirm appointment times, quote prices, and read back reference numbers, “the model will usually say approximately what you told it to” is not an acceptable contract.
Secondary Problems
| Problem | Evidence |
|---|---|
| Tool-call reliability worse | LiveKit: realtime models decide from audio, cascaded pipelines decide from clean text. Noisy input → noisier tool decisions. |
| Interruption selectivity worst-in-class | τ-voice measured OpenAI at 6% selectivity — it responds to nearly all backchannels and non-directed speech. A caller saying “mhm” to someone else in the room derails the call. |
| 8kHz is structurally hostile | Realtime models train on 16–48kHz web audio. G.711 narrowband is out of distribution — see voice-agent-stt-vendor-landscape-2026-07-21. |
The interruption number deserves emphasis: 6% selectivity means the model treats ~94% of incidental speech as a turn it should react to. On a phone line with background noise, TV, or a second person in the room, that is a broken call.
Counter-Evidence (Recorded Honestly)
Sierra ships GPT-Realtime-2 in production and it is genuinely faster
Sierra measured, against their own cascade:
- ~30% faster P50
- up to 200% faster P90
P90 is the percentile that actually matters, so this is not a trivial win.
But read what they had to build to get it. Sierra only achieved this by constructing “an agent harness that provides the control plane,” consisting of:
- a supervisor layer
- PII redaction
- tracing
- and custom-tuned VAD replacing the model’s built-in turn detection
That last item is the tell. They rebuilt a cascade’s worth of control plane around the S2S core — including throwing out the turn-detection that is supposedly one of S2S’s structural advantages. The latency win is real but it is not available to someone adopting the Realtime API as a drop-in; it’s available to someone who builds the same control plane a cascade requires and then swaps the core.
Given that this project is building that control plane in Rust anyway, S2S is worth revisiting later as a swappable core — but not as the v1 architecture, and not before the alphanumeric and language-drift failure modes are independently retested.
Cost Is No Longer the Argument
Gemini Live reached rough parity with a cascade
Option $/min Gemini Live ~$0.023 — rough parity with a cascade Cascade (this project) 0.020 is STT+TTS OpenAI gpt-realtime-2.10.123 Cost no longer disqualifies S2S generally. It still disqualifies OpenAI’s S2S.
Two extra points:
- OpenAI’s cost scales with conversation length — full context is re-sent each turn, so a long call costs superlinearly. The $0.073–0.123 range is the observed range, driven by call length.
- Gemini Live has hard blockers of its own despite the good price:
- PCM16 16k/24k only — no μ-law. Every frame needs transcoding in both directions, adding latency and code, on a carrier that only speaks μ-law 8k (voice-agent-telephony-carriers-2026-07-21).
- 15-minute audio session limit and ~10-minute WebSocket lifetime. For a receptionist this means building session-resumption machinery for calls that run long — plus a visible seam mid-call.
Verdict
| Requirement | Cascade | S2S |
|---|---|---|
| Read back a booking reference correctly | ✅ | ❌ ~70% |
| Not switch language on hearing a name | ✅ | ❌ first-turn failure |
| Say exactly what it was told to say | ✅ | ❌ no verbatim guarantee |
| Transcript == what caller heard | ✅ | ❌ |
| Ignore backchannels | tunable | ❌ 6% selectivity |
| μ-law 8k native | ✅ | ❌ (Gemini PCM-only) |
| Latency | ~1,048ms measured | 30% / 200% better with a full custom harness |
Cascaded pipeline. Revisit S2S as a swappable core once the Rust control plane exists.
Sources
- Developer report:
MAR17Malphanumeric corruption, ~70% accuracy, cascade comparison [measured, single reporter] - Sierra τ-voice benchmark, arXiv 2603.13686 — authentication bottleneck, interruption selectivity [measured]
gpt-realtime-2.1language-drift report, 2026-07-21 [user report]- Gemini Live over Twilio language-drift report [user report]
- OpenAI Realtime prompting guide — instruction-wording caveat [vendor]
- xAI
force_messageprimitive [vendor] - LiveKit — audio vs clean-text tool-call decisioning [vendor]
- Sierra production writeup — GPT-Realtime-2 harness, P50/P90 deltas [vendor, self-reported]
- Gemini Live / OpenAI Realtime pricing and session limits [vendor]