An AI feature that works in a demo is easy now. One that keeps getting better after launch is still rare. This small-group workshop is about building that second kind: systems that learn from their own results, improve on a schedule, and never change without passing a test and getting a person's sign-off.
It follows our self-improving voice agents workshop, but the ideas apply to anything: support bots, coding agents, research assistants, sales tools, internal workflows.
The loop every self-improving system needs
- Define "good" first. Write a rubric and collect 20 to 50 real examples with known right answers. No evaluation set, no improvement: you'd just be changing things.
- Capture signals. Explicit ones (thumbs up or down, ratings) and the more honest implicit ones: did the user edit the output, retry, abandon, or complete the task?
- Reflect. On a schedule, a reviewer agent reads recent failures, groups them and names the likely cause.
- Propose. It drafts one concrete change, with the evidence for it.
- Gate. The change runs against the evaluation set. It ships only if it beats the current version and breaks nothing that already worked.
- Approve and version. A person signs off, the change gets a version number, and rolling back is one click.
What the system is allowed to change
Being clear about this up front is most of the safety work.
- Safe to evolve: instructions and prompts, worked examples, memory and knowledge, tool descriptions, which model handles which kind of request.
- Fixed by people: permissions, spending limits, anything touching money or customer data, the evaluation set itself, and the rules for what counts as "good."
Patterns from Charlotte builders
Some of the best examples of this have come from pitches at our own meetups:
- Adversarial scoring before code. At Bloxx, every plan an agent writes is scored 1 to 10 by a separate critic agent on architecture, code quality and customer value, and has to back up its claims with evidence from the codebase before a person reviews it.
- Get grilled before you build. At our November meetup, the Gridramp team showed a "grill me" step where the model interviews them one hard question at a time until an idea holds up.
- Draft, then approve. Gridramp's in-product AI builds a whole project in a draft, tracks every change, and nothing goes live until a person publishes it. That's the same gate a self-improving system needs.
Common ways this goes wrong
- Optimizing the metric, not the outcome. If the grader rewards long answers, you'll get long answers. Revisit the rubric as often as the prompts.
- Overfitting to recent complaints. Fixing last week's loudest user can quietly break the common case. That's what the fixed evaluation set is for.
- Silent drift. Models get updated under you. Re-run your evaluations whenever the underlying model changes, even if you changed nothing.
- Runaway costs. Reflection and testing cost tokens. Run them on a schedule and on samples, not on every request.
What you'll leave with
- A rubric and a starter evaluation set for something you're building.
- A working loop: a log, a grader, an improver and a test gate.
- A checklist for what your system may and may not change on its own.
Questions
When is it?
We're scheduling it between monthly meetups. Join the mailing list and you'll hear first, since spots in small groups are limited.
Do I need to have built an agent already?
No. Bring an idea or something half-built. Groups are split by experience.
Which tools will we use?
Whatever you already use. The loop is tool-agnostic, and we'll show it with common models and coding agents.