How We Bake Off Our AI Stack
Every AI engine we run in production — the voices that answer phones, the transcription that reads calls, the models that reason about requests — had to win a head-to-head competition to get there. This is how the competition works, and why it matters to your business.
The problem with picking AI by press release
A new “best-ever” AI model ships roughly every week. Some of them really are better. Most are better at something, worse at other things, and all of them cost time and money to adopt. If you pick by press release, you end up paying twice: once for the migration, and again when the shiny new engine turns out to stumble on the one thing your business actually needs it to do.
Our clients don't experience “models.” They experience a phone line that answers naturally, a transcript that's accurate enough to act on, a request that gets handled correctly the first time. So when we decide what runs in production, we don't ask which engine has the best demo. We make the candidates compete on our work, on our test bench, under identical conditions — and we measure. Internally we call these bake-offs, and every AI capability we operate goes through one.
In one sentence: same inputs → competing engines → automated scoring → a winner with numbers behind it → and a scheduled retirement for the loser.
The method: five steps, no exceptions
- Same inputs. Every candidate gets an identical, fixed test set drawn from real work — the same scripted lines for a voice engine, the same (anonymized) call recordings for transcription, the same task battery for a reasoning model. No cherry-picking, no vendor-supplied demo data.
- Head-to-head rendering. The engines run side by side in our GPU lab, on the same hardware class they'd run on in production. We capture every output and every wall-clock timing, per item, for both engines.
- Automated scoring. Outputs are graded by the measure that fits the job: perceptual quality suites for synthesized speech, word-error-rate for transcription, task rubrics plus latency for reasoning models. Human review confirms the scores; it doesn't replace them.
- Kill switches. Nothing ships without a way back. A challenger is promoted behind a switch we can flip in minutes, and it runs shadow or partial traffic before it runs all of it. If the numbers regress in the real world, we revert — same day, no drama.
- Losers retire. A bake-off ends with a decision, not a shelf of half-adopted tools. The losing engine gets a retirement date, its licensing gets a final review, and its models come off our production fleet. That discipline is where the cost savings actually live.
The pipeline every engine runs: same inputs, head-to-head render, automated scoring, human confirmation — then promotion behind a kill switch, or retirement.
Four bake-offs running right now
This isn't a slideware process — here's the current slate on our evaluation cluster. Results, rankings and winners land in Part 2.
Voice synthesis: incumbent vs. challenger
Our production voice-cloning engine defends its title against a newer challenger. Both render the same ten evaluation lines across three distinct voices — identical scripts, identical reference audio — and we time every line as it renders.
Scored by: TTSDS2 perceptual suite + human listening + per-line latencySpeech-to-text: three-way
Three transcription engines — Parakeet, Whisper and Canary — compete on real, anonymized call recordings from production phone lines, accents and hold music included. The test set is the job, not a lab benchmark.
Scored by: word error rate (WER) + throughputLanguage models: the nightly scoreboard
Candidate reasoning models run a bounded battery of real task types every night on our evaluation harness, emitting machine-readable scores. New releases earn their way onto the board; nobody gets grandfathered in.
Scored by: task rubric + latency + cost per taskPhone-agent brains: hosted vs. local
For the AI that holds a live phone conversation, we route between a frontier hosted model and a locally-served open model — and let the rubric and the stopwatch decide which handles which calls. A voice agent that thinks for four seconds has already lost the caller.
Scored by: conversation rubric + time-to-first-wordCase study in miniature: replacing a phone voice
A concrete example from the voice program. One of our clients runs a call-intake line where the agent's voice is the first impression. When a promising new voice engine appeared, we didn't swap it in to see what happens. We rendered the same ten lines — greetings, questions, numbers, long sentences — in three different voices on both the incumbent and the challenger, timed every render, and pushed all of it through an automated perceptual scoring suite plus human ears. The engine that wins gets the phone line. The one that loses gets a retirement date. The client never hears the bake-off — they just never hear a bad voice.
The scoreboard is running now
Latency charts, quality scores and the full rankings table land in Part 2 — the current slate is rendering on our evaluation cluster as this post goes live.
What this buys you
- Quality you can hear (and read). The voice on your phone line, the transcript in your inbox and the answer to your request all won a measured competition. When something better genuinely exists, it gets adopted — on evidence, not enthusiasm.
- Costs that go down instead of sideways. When a locally-run open model matches a premium hosted one on a given job, the bake-off proves it — and that job moves to the cheaper engine. When it doesn't, we don't pretend. Either way you're paying for measured capability, not brand names. And because losers actually retire, you're never funding two engines to do one job.
- Reliability by design. Every promotion ships with a kill switch and a revert path. New engines earn traffic gradually. Licensing is reviewed before adoption and again at retirement — so nothing in the stack is a legal or operational surprise.
- Speed without recklessness. Because the harness already exists, evaluating a new model takes days, not quarters. We move fast because we measure — not instead of measuring.
The lifecycle, start to finish
Each bake-off program runs the same arc on our internal board: Discovery (candidates render head-to-head), Ingest (evaluation data is prepared and cleaned to a repeatable standard), Evaluation (automated scoring, e.g. the TTSDS2 suite for voice), Streaming readiness (the winner has to hold up under live, real-time conditions — a phone call doesn't wait), and Migration (promotion with kill switches, licensing review, and the loser's retirement). Five stages, every engine, every time.
Coming in Part 2: the results — the full scoreboard from the current slate, which engines won, which got retired, and what changed on our production floor because of it.
Want AI that's measured, not marketed?
The same evaluation discipline behind our own stack is available to your business — phone agents, transcription, automation, all proven before they touch your customers.
Talk to us →