Voice Agents in Production: What We Learned

A voice agent that sounds great in a demo can still fall apart on a real phone line. Callers interrupt, talk over background noise, change their minds and ask things nobody planned for. Here are the lessons we've learned taking voice agents from prototype to production calls.

Latency Is Everything

In conversation, a pause of more than about a second feels awkward. Every step adds delay: speech-to-text, the language model, any tool calls, and text-to-speech. We set a latency budget for each step and measure it on every call.

• →Stream everything: start transcribing while the caller speaks and start speaking while the reply is still being generated.
• →Use fast models for the conversation and save heavier models for background tasks.
• →Fill gaps naturally: a short "Let me check that for you" covers a calendar lookup.

Handle Interruptions

People interrupt. If the agent keeps talking over them, the call feels robotic. Good voice agents stop speaking the moment the caller starts, listen, and respond to what was just said. Tuning this barge-in behaviour takes testing with real recordings, not just scripts.

The best voice agent is the one callers don't notice is an agent.

Design the Handoff

Not every call should stay with the agent. Define clear rules for when to transfer to a person: complaints, complex requests, or a caller who simply asks for a human. When it transfers, it should pass on a short summary so the caller doesn't repeat themselves.

Connect to Real Systems

A voice agent is only useful if it can act. Most of the value comes from integrations: checking calendar availability, creating CRM records, sending confirmation texts. Build these with timeouts and fallbacks, so a slow system never leaves a caller in silence.

Test With Real Calls

• 1Record and review a sample of calls every week.
• 2Track key numbers: resolution rate, transfer rate, average call length and caller satisfaction.
• 3Turn failures into test cases so the same problem never happens twice.
• 4Test accents and noise from your real customer base, not just a quiet office.

Voice agents can answer every call, day and night, but only if they're built for real conversations. That's the standard Devex Hub holds every voice deployment to.