How to Build a Voice Agent: A Practical 2026 Guide

A voice agent is software that listens to a caller, decides what to say or do, and talks back. It needs speech-to-text, a language model and text-to-speech (or one speech-to-speech model), plus a phone number or web widget. The fastest way to build one is a hosted platform such as Vapi or Retell AI.

Core partsListen, think, speak

Fastest routeA hosted voice agent platform

Hardest partLatency and turn-taking

How a voice agent works

There are two designs. In a pipeline, three models run in a row: speech-to-text turns the caller's audio into words, a language model writes the reply, and text-to-speech reads it out. In a speech-to-speech setup, one realtime model takes audio in and sends audio out.

The trade-off is control versus simplicity. A pipeline lets you pick and swap each part and gives you a text transcript at every stage, which makes debugging easier. A speech-to-speech model has fewer moving parts and fewer hand-offs between services, but you see less of what happened inside a turn.

Four other pieces sit around the models:

Telephony. A carrier gives you a phone number and carries the call audio to your agent, usually over SIP or a media stream. Web and app agents skip this and use the browser microphone.
Turn detection. Something has to decide the caller has finished talking. Wait too long and the agent feels slow. Jump in too early and it cuts people off.
Interruptions. When the caller talks over the agent (often called barge-in), the agent should stop speaking and listen.
Tools. Functions the model can call during the conversation, such as checking a calendar, looking up an order or transferring the call.

Three ways to build one

Route	Best for	You manage	Examples
Hosted platform	A first agent, non-engineers, shipping in days	Instructions, tools, voice and phone number settings	Vapi, Retell AI, ElevenAgents, Bland, Synthflow
Open-source framework	Developers who want to choose every model and own the code	Your code, model accounts, hosting and scaling	LiveKit Agents, Pipecat
Direct realtime API	Custom apps built around one provider's speech-to-speech model	The audio connection, conversation state, tools, and telephony if you need it	OpenAI Realtime API, Gemini Live API, xAI's speech-to-speech API

If you are not sure, start on a hosted platform.

Build your first voice agent in 8 steps

Pick one narrow job. "Answer the phone for a dental office after hours and book cleanings" is a good first job. "Handle all our calls" is not.
Write the instructions and a greeting. Say who the agent is, what it can and cannot help with, and what to do when it is unsure. Write the first sentence it speaks word for word, keep it short, and say up front that the caller is talking to an AI assistant.
Choose the route. Use the table above. For a first build, a hosted platform means you can make a test call the same day and worry about infrastructure later.
Give it one or two tools. Start with one lookup (hours, availability, order status) and one action (book a slot, take a message). Describe each tool in plain words so the model knows when to use it, and tell it what to say while the tool runs.
Connect a phone number or web widget. Most platforms let you buy a number inside the product or bring one from a carrier such as Twilio or Telnyx.
Test with real calls and edge cases. Call it yourself, then hand the number to people who did not build it. Try unusual names, dates said three different ways, interruptions, long silences, a noisy car and a caller who changes their mind halfway through.
Add a human hand-off and a fallback. Decide when the agent transfers to a person, and what happens when nobody picks up (take a message, send a text, promise a callback).
Log every call and review it. Keep transcripts, tool calls and outcomes, and read or listen to a batch every week.

What makes voice agents feel good or bad

Latency

On a phone, silence reads as a broken line. The delay you feel is the sum of every stage: detecting the end of speech, transcription, the model, any tool call, and the first bit of audio back. Measure it per stage so you know which part to fix.

Turn-taking and barge-in

People pause mid-sentence, say "um" and talk over each other. Tune how long the agent waits before replying, and make sure it stops talking when interrupted.

Reading critical details back

Names, phone numbers, dates, addresses and amounts get misheard. Have the agent repeat them and ask for a yes before it books or saves anything.

Short replies

Text that looks fine on a screen is too long out loud. One or two sentences per turn, one question at a time, no lists of six options.

Handling "I didn't catch that"

Give the agent a way to recover that is not the same sentence on repeat. Ask again in different words, offer a choice, and after two failed tries move to a hand-off.

Knowing when to transfer

Write the transfer rules down: the caller asks for a person, the caller is upset, the topic is outside the agent's job, or the agent has failed twice.

Tools to know

These are common building blocks, each described from its own site or docs. Pricing is usually per minute of conversation, sometimes with separate charges for the model, the voice and the phone line, so check each vendor's pricing page (for example Vapi, Retell AI, Deepgram and Twilio) before you commit.

Hosted platforms

Vapi: a developer platform for voice agents, with assistants, tools, phone numbers and web calls.
Retell AI: a platform to build, test, deploy and monitor AI voice and chat agents.
ElevenAgents: ElevenLabs' agents platform, with web widgets, SDKs, SIP trunks and a Twilio connection.
Bland: a platform to build, monitor and improve voice agents and assign them phone numbers.
Synthflow: a platform for voice and chat agents with an agent editor and a flow designer.

Open-source frameworks

LiveKit Agents: an open-source framework for Python and Node.js with turn detection, interruption handling, telephony support and both pipeline and realtime models.
Pipecat: an open-source Python framework for voice and multimodal agents, from the team at Daily, that works with many AI services.

Realtime speech-to-speech APIs

OpenAI Realtime API: speech-to-speech models that work directly with audio and can call tools, over WebRTC or WebSocket.
Gemini Live API: Google's realtime voice and vision API, with barge-in and function calling.
xAI speech-to-speech: a realtime voice agent API over WebSockets with tool use.

Speech-to-text and text-to-speech

Deepgram: speech-to-text, text-to-speech and a Voice Agent API.
AssemblyAI: streaming speech-to-text aimed at voice agents, plus a text-to-speech API.
Cartesia: Sonic for text-to-speech and Ink for streaming speech-to-text.
ElevenLabs: text-to-speech voices and speech recognition, also used inside its agents platform.

Telephony

Twilio: Programmable Voice for making and receiving calls, plus ConversationRelay for connecting AI agents to calls.
Telnyx: a carrier with a Voice API and phone numbers, plus its own voice AI agents product.

Last checked November 10, 2026. Details change, so confirm on the vendor's page before you build. Spot something out of date? Tell us.

Before you put it on a real phone line

Heads up: rules on recording calls and on using AI voices vary by state and country, and some places require every person on the call to consent to recording. The FCC's consumer guide points to state law for recording, and the FCC has ruled that calls made with AI-generated voices count as "artificial" under the Telephone Consumer Protection Act. This is not legal advice. Check the rules where your callers are, and be extra careful with outbound calls.

Data handling. Decide what you record, where transcripts and audio are stored, how long you keep them and who can read them. Do not collect card numbers or health details by voice unless you have set up for it properly.

A short test checklist:

The greeting says it is an AI assistant and mentions recording if you record.
Ten or more calls from people who did not build it, on real phones.
Names, dates and numbers are read back and confirmed.
Interrupting the agent stops it within a word or two.
The transfer works, including when nobody answers.
A tool failure produces an apology and a fallback, not dead air.

Next step: make it improve itself

Once your agent is taking calls, the logs become the raw material for making it better. Our free workshop on self-improving voice agents covers that loop hands-on at the Wednesday, December 2, 2026 meetup at Hygge Coworking in Charlotte (RSVP on Meetup, or see all events). The follow-up, self-improving AI systems, applies the same idea beyond voice.

FAQ

How long does it take to build a voice agent?

A working demo on a hosted platform can take an afternoon. Getting it ready for real callers takes longer, because most of the work is testing edge cases, tuning turn-taking and setting up hand-offs.

Do I need to code to build a voice agent?

Not for a first version. Hosted platforms let you write instructions, pick a voice and attach a phone number in a dashboard. You will need code, or an AI coding tool, for custom tools and for open-source frameworks such as LiveKit Agents or Pipecat.

What does a voice agent cost to run?

Most vendors charge per minute of conversation. The total usually combines a platform fee, the language model, speech-to-text, text-to-speech and telephony, plus a monthly fee for the phone number. Rates change often, so check each vendor's pricing page.

What is the difference between speech-to-speech and a pipeline?

A pipeline chains three models: speech-to-text, a language model and text-to-speech. A speech-to-speech model does it in one step, audio in and audio out. Pipelines give you more control and visibility, and speech-to-speech has fewer parts to manage.

Is it legal to record AI phone calls?

It depends on where you and your callers are. Recording rules vary by state and country, and some require consent from everyone on the call. Tell callers at the start that they are talking to an AI and that the call is recorded, and get legal advice for your situation.

How to build a voice agent