AI Bake-Off, Part 2: The Results
In Part 1 we explained how we make AI engines compete for a place in production — same inputs, head-to-head, scored by machine and confirmed by ear. Here are the results: who won, the numbers behind it, and what actually changed on our production floor.
Three of those contests have finished. A new voice engine challenged our incumbent. Three transcription engines competed to correctly hear specialized call vocabulary. And a field of AI “brains” competed to hold a live phone conversation without keeping the caller waiting. Every result below comes with a number — and, in the honest tradition of measuring things, a catch worth reading.
The short version: in all three contests a leaner, cheaper option won or matched a heavier one — but only the measurement proved it, and in every case a single metric would have told us the wrong thing.
A zero-shot voice engine unseats our hand-tuned one
Our incumbent is a voice-cloning engine we tune by hand, one voice at a time — a process that takes days per voice. The challenger, an open zero-shot engine (CosyVoice 3), promises the same result from a single reference clip with no tuning at all. We rendered the same ten lines across three distinct voices on both engines and scored them three ways: how much the clone sounds like the target speaker, how accurately it pronounces the words, and a full perceptual quality suite (TTSDS2).
Winner: the zero-shot challenger. It matched or beat our hand-tuned engine on speaker identity for all three voices, and won or tied overall on every one.
3 / 3
voices where the challenger won or tied overall
.70
challenger speaker-similarity on the hardest voice — vs .34 for the hand-tuned clone
~1 hr
to stand up a new voice — down from days of hand-tuning
The biggest win came on the voice whose hand-tuned clone had always been weakest: the zero-shot engine roughly doubled the speaker-similarity score. On speed, there's a teaching moment worth admitting to. Our first measurement had the challenger rendering twelve times too slowly — which turned out to be a misconfiguration quietly running it on the CPU instead of the GPU. Once fixed, it renders a line in about 3.8 seconds (comfortably faster than real time). We report that because the lesson of this whole program is to measure honestly, including when the first number embarrasses you.
⚠️ The catch worth readingOn one voice, the word-accuracy metric said our incumbent garbled about 22% of the words — yet the perceptual suite rated that same clip as more intelligible. Both can be true: sounding like natural, fluent human speech is not the same thing as getting every word right. It's exactly why we never let one metric decide — we use several, plus a human ear.
A two-line change takes transcription to a perfect score
Three transcription engines ran across thirty real, anonymized call recordings from a client's phone line. The metric that mattered here wasn't raw accuracy on every word — it was domain-vocabulary recall: does the engine correctly hear the specialized terms the business actually depends on? A transcript that nails “hello” but mangles the name of the service is worse than useless.
Winner: Whisper, with a domain-vocabulary prompt. Feeding the engine a short list of the terms it should expect took recall from 87% to a perfect 100% — a two-line change.
That small hint completely fixed a persistent problem: the engine had been reliably mis-hearing the line's specialized greeting, and the prompt eliminated it. A lightweight, very fast engine (Parakeet) scored lower on recall but earns a permanent role as our low-latency fallback for when speed matters more than catching every rare term. A third engine (Canary) ran, but needs extra long-form handling before it can compete — not adopted this round.
⚠️ The catch worth readingThe winning fix wasn't a more expensive model or more computing power — it was telling the engine what words to expect. The prompt fix ships regardless of which engine you run, and it's the kind of cheap, high-leverage change you only find when you measure the thing your business actually cares about instead of a generic benchmark.
The AI that answers the phone: fast-and-local wins the everyday calls
For the model that holds a live phone conversation, quality and speed are both non-negotiable. We scored candidates on a 0–5 rubric — empathy, accuracy, respecting hard guardrails (never give legal or medical advice, never promise an outcome), and whether it actually moves the caller forward — with an automatic fail for any answer that crossed a guardrail. Then we timed every response, because a caller won't wait.
Winner (for most calls): our fast local model. A compact mixture-of-experts model scored ~4.0 on quality at just 1.7 seconds per turn — nearly matching the frontier hosted model's quality at a fraction of the wait, and 24× faster than the heavyweight local model.
1.7s
our local model's response time — vs 6.8s frontier, 40.5s for the dense 32B
~4.0
local model quality (0–5) — against 4.4 for the frontier model
24×
faster than the heavyweight local model it beat on speed
The heavyweight local model actually reasoned well (3.7), but at roughly 40 seconds per turn it was never going to hold a phone call — the stopwatch disqualified it before the rubric mattered. So the decision isn't “pick one brain,” it's route between them. The fast local model now handles the everyday turns in-house — greetings, benefits questions, scheduling — and on one scenario it out-scored every model in the field. The frontier model is held in reserve for the four hardest turn-types, where nuance is worth the extra couple of seconds: legal-deadline questions, deflecting medical questions, handling wrong numbers gracefully, and non-English callers.
⚠️ The catch worth readingThe scoring caught something no demo would have: two different models occasionally read their private “thinking” out loud to the caller — one defect in each model family — a flaw that stays invisible until you actually grade the transcripts. That's the whole thesis of this program in one bug: you cannot fix what you haven't measured.
What changed on our production floor
A bake-off isn't a blog post — it ends with something different running for our clients. Here's what these three results actually moved:
Voice: zero-shot by default
We now stand up a new phone voice from a single reference clip in about an hour, and only fall back to hand-tuning when the zero-shot result falls short. Days of work become an afternoon — at equal or better quality.
Transcription: the prompt ships
Whisper with a domain-vocabulary prompt is the new default, at 100% recall on the terms that matter, with the fast engine kept as a low-latency fallback. The two-line fix shipped regardless of engine.
Phone brain: local first
Most calls now stay on our fast local model — cheaper and quicker — with the frontier model reserved for the handful of turn-types where nuance earns its keep. Cost goes down; quality doesn't.
Notice the pattern. In every contest, the winning move was the leaner one — a zero-shot engine, a two-line prompt, a compact local model — and in every contest a single metric, taken alone, would have pointed us somewhere worse. Speech that scored badly on word-accuracy sounded better. A perfect transcript came from a prompt, not a bigger model. A brilliant reasoner was useless on a phone because of a stopwatch. This is why we run bake-offs: not to chase the biggest model, but to find the cheapest thing that measurably does the job — and to catch the defects that only show up under a real test.
The through-line: measurement doesn't just pick winners — it keeps you from paying more for less. Every one of these decisions lowered cost or latency without giving up quality, because we let the numbers, not the marketing, make the call.
Want AI that's measured, not marketed?
The same evaluation discipline behind our own stack is available to your business — phone agents, transcription, automation, all proven on your real work before they touch your customers.
Talk to us →