Voice AI Agents: Latency Is the Whole Product
The models sound human already. Nobody abandons a call because the voice was unconvincing — they abandon it because of the pause before the answer.
Ritik MakhijaTL;DR
Voice AI agents have roughly an 800ms end-to-end budget — transcription, reasoning, tool calls and speech generation all share it — before a conversation starts feeling broken. Speech quality is solved; latency, turn-taking and interruption handling are not, and neither is the integration work behind the voice. An agent that sounds perfect but can't look up an order is an expensive phone menu.
Key takeaways
- The end-to-end latency budget is roughly 800ms; past that, callers talk over the agent.
- Speech quality is a solved, purchasable problem — latency and turn-taking are not.
- Barge-in handling is what separates a conversation from a phone tree.
- Mishearing compounds: bad transcription becomes confidently wrong reasoning.
- Most voice projects fail on integration, not on voice.
Every voice AI demo in 2026 sounds great. The speech models are genuinely excellent — natural prosody, convincing pauses, the lot. And it doesn't matter much, because nobody has ever abandoned a call because the voice wasn't convincing enough. They abandon it because of the silence before the answer.
The 800 millisecond budget
In text, a two-second pause is invisible — you don't even notice a chat agent thinking. In speech it's an eternity. Human conversation runs on gaps of roughly 200ms between turns; stretch that past about 800ms and the caller assumes something broke and starts talking again, over the agent, which then has to handle the interruption it caused.
The brutal part is that this budget is shared. Transcription, the model's reasoning, any tool calls it makes, and speech generation all have to fit inside it together:
| Stage | Rough budget | What eats it |
|---|---|---|
| Speech-to-text | ~100–150ms | Waiting for end-of-speech detection |
| Reasoning | ~200–300ms | Model size, prompt length |
| Tool calls | ~100–300ms | Your API — the usual culprit |
| Text-to-speech | ~100–150ms | Time to first audio, not total |
Notice which row is yours. Speech-to-text and text-to-speech are vendor problems with vendor solutions. The tool call into your CRM is the one nobody budgets for, and it's routinely the slowest thing in the chain.
Turn-taking is harder than it looks
Knowing when the caller has finished speaking is genuinely hard. People pause mid-sentence to think. They say 'um'. They trail off. End-of-speech detection that's too eager cuts them off; too patient and you've spent your whole latency budget waiting for someone who already finished.
And they interrupt. An agent that can't be interrupted mid-sentence — barge-in, in the jargon — feels exactly like the phone tree the caller was hoping to escape. Handling it means stopping speech generation instantly, working out what was actually heard versus what was said, and picking up from the right place.
Mishearing compounds
In text, a typo is visible and self-correcting. In voice, a wrong transcription becomes wrong reasoning becomes a confidently wrong answer, and the caller has no scrollback to check against. There's no way to notice the mistake until the consequence arrives.
The fix is unglamorous: treat low transcription confidence as a first-class state and ask, rather than guess. "I heard Smith — is that right?" is a far better experience than confidently looking up the wrong account.
The part that actually fails
A voice agent that can talk but can't look up an order is a very expensive phone menu.
Here's the thing nobody puts in the demo: the voice layer is the smaller half. Speech-to-text, reasoning and text-to-speech are all purchasable. What isn't purchasable is the agent underneath — the tool calls into your CRM, retrieval over your knowledge base, guardrails, escalation that carries context, and evaluation on real recordings.
That layer is ordinary agent engineering, and it's where voice projects actually die. Which is also why I build voice AI agents on existing voice infrastructure rather than reinventing transport — the budget belongs on the part that's yours.
How to know it's working
- Containment rate — calls resolved without a human. The headline number.
- Escalation quality — did the handoff carry context, or did the caller repeat themselves?
- p95 latency — the average is a lie. The slow tail is what callers remember.
- Transcript review — against real calls, not scripts. Voice is unforgiving and 'it sounded fine' isn't a number.
Thinking about a voice agent? Tell me the call type eating the most time and I'll tell you whether it can hold it — and where it should escalate instead.
Frequently asked questions
Human conversation runs on gaps of roughly 200ms between turns. Stretch that past about 800ms and callers assume something broke and start talking again, over the agent. The budget is shared across transcription, reasoning, tool calls and speech generation — so every stage has roughly 100–300ms, and your own API call is usually the slowest link.
Barge-in is the caller interrupting the agent mid-sentence. Handling it means stopping speech generation instantly, reconciling what the caller actually heard against what was said, and resuming from the right place. An agent that can't be interrupted feels exactly like the phone tree callers were trying to escape.
No — speech synthesis in 2026 is genuinely excellent and purchasable from several vendors. Nobody abandons a call because the voice was unconvincing. The unsolved problems are latency, turn-taking, interruption handling, recovering from mishearing, and integrating with the systems that hold your data.
Containment rate (calls resolved without a human), escalation quality (whether the handoff carried context), p95 latency rather than average, and transcript review against real recordings. Averages hide the slow tail, and the slow tail is what callers actually remember.
About the author
Ritik Makhija
Founder & Product Lead · AI Kaptan
I build AI agents and automation that run in production — and I've open-sourced 5,000+ workflows so you can read the work rather than take my word for it. I run outreach infrastructure sending 6,000 emails a day on this stack, and I've mentored 700+ builders 1:1.
Got a process you're trying to automate?
Tell me what it is and I'll say straight whether it needs an agent, a plain workflow, or nothing at all. Free 30 minutes, no pitch.