Most voice agents ship once and then stay exactly as good, or as bad, as launch day. In this workshop we build one that gets better every week: it answers calls, grades its own conversations, and proposes fixes that you approve before they go live.
It runs as the opening workshop at our December meetup. You don't need to be an engineer. We split into small groups by experience, so first-timers and people already running agents in production each get something useful out of the hour.
What you'll build
A phone or browser voice agent for a made-up local business, such as a front desk that answers questions and books appointments, plus the loop around it that turns every call into an improvement:
- A working voice agent with clear instructions, a couple of tools (look up hours, book a slot) and a hand-off to a human.
- A call log that captures the transcript, every tool call, the outcome and how long each reply took.
- A grader that scores each call against a short rubric.
- An improver that reads the failures and drafts a change: better instructions, a new example, a clearer tool description.
- A test gate that replays past calls against the change before you approve it.
How a voice agent works
Every voice agent does three jobs on every turn of the conversation:
- Listen. Speech-to-text turns the caller's audio into words and works out when they've finished talking.
- Think. A language model reads the conversation, its instructions and its tools, then decides what to say or do next.
- Speak. Text-to-speech turns the reply back into audio.
Some newer models handle all three in one step (speech-to-speech), which cuts delay but gives you less to inspect. Either way, the things that make or break a voice agent are rarely the model itself. They're speed (a pause much over a second feels broken on a phone), turn-taking (letting callers interrupt), and how well the agent recovers when it mishears a name or a date.
What makes it "self-improving"
Every call is free training data, and most teams throw it away. A self-improving agent runs a loop:
- Record. Save the transcript, tool calls, outcome (booked, transferred, hung up) and reply times for every call.
- Grade. Score each call with simple checks (did a booking happen?) plus a model acting as a judge against a written rubric: accurate, polite, stayed on task, knew when to hand off.
- Find patterns. Group the low-scoring calls. One bad call is noise; twelve callers confused by the same question is a fix.
- Propose. An improver agent drafts a specific change to the instructions, examples, tool descriptions or knowledge base, and explains which calls it should fix.
- Test. Replay a fixed set of past calls, or simulated callers, against the new version. It has to fix the target problem without breaking anything that already worked.
- Approve and ship. A person reviews the change and the test results, then publishes it as a new version you can roll back.
Guardrails we'll cover
- Lock the test set. If the improver can edit the tests, it will eventually grade itself an A.
- Check the judge. Spot-check the grader's scores against your own every so often, because judges drift too.
- Separate what can change from what can't. Wording, examples and knowledge can evolve. Permissions, payment steps and safety rules stay fixed.
- Respect callers. Tell people calls are recorded, and strip personal details from anything you keep for training.
What to bring
- A laptop and an API key for any major model provider (we'll have a couple of shared options if you don't).
- Optional: an account on a voice platform you already like. We'll show a vendor-neutral setup, so nothing is required.
- One real use case you'd like a voice agent for. The best groups work on something they'd actually ship.
Questions
Do I need to know how to code?
No. Beginner groups build with no-code and AI coding tools; experienced groups go deeper on evals, latency and the improvement loop.
Is the workshop separate from the meetup?
No, it's the first part of the Dec 2 meetup. Doors open at 6:30 PM, the workshop runs first, then two builder pitches and networking until 8:30.
Will there be a follow-up?
Yes. We're planning a longer small-group session on building self-improving AI systems more generally. Join the mailing list to hear when it's scheduled.