Why one engine isn't enough: more engines only help if they fail differently

We ran a comparison on a 42-minute business interview. One transcript came straight out of ElevenLabs. The other went through our full pipeline. The two were 91% identical.

The remaining 9% was exactly the part researchers care about most.

"Mostly right" is the dangerous part

We picked out 19 key terms from the interview: brand names, model numbers, place names, industry jargon. In the straight-out-of-the-engine transcript, 10 were wrong, 4 were partly wrong, and 5 were right. One brand name came up six times and was never once written correctly. Each time it became an ordinary word that sounded similar and meant something else entirely.

To be fair to ElevenLabs, it did well that day. Nothing collapsed, nothing was dropped, and it got the speakers right about 98% of the time. Judged as a whole, you'd struggle to fault it.

That's the problem. A reader who sees a fluent sentence has no reason to doubt the noun in the middle of it. And the three other engines we had on hand got those same terms wrong as well. So the point isn't that one engine is bad. It's that any engine on its own will mishear industry terms it hasn't come across before.

Our own "optimisation" was just shuffling the deck

Before sending audio to an engine, we split it into sections and normalise the format. We assumed this would improve accuracy.

It fixed 22 things and broke 25. Roughly a coin toss.

We eventually worked out why. This engine is very sensitive to where in the audio it starts listening. The same model number came out as "X6" when the section started at zero seconds, and as "X-six" spelled out when it started at 1:30. Even the unprocessed transcript wasn't consistent: "X-six" for the first 32 minutes, "X6" after that.

Getting there took three controlled experiments, each ruling out an obvious explanation: it wasn't the length of the file, it wasn't the audio format, and it wasn't the language setting. If we'd stopped at the first plausible answer, we'd have written the wrong cause into our own documentation.

What actually improved the transcript was engines checking each other

Three other engines transcribe the same audio, and we compare all of them together, with a glossary of terms. Of the 25 things our splitting had broken, 14 were recovered, or 56%.

The model number sorted itself out the same way. Another engine wrote "X6" every single time, so whenever the main engine slipped, the final transcript still had the right form to draw on.

Where the engines can't settle it, we don't decide for the reader. We flag the spot as uncertain so someone can listen to that second. This interview had 7 flags.

More engines isn't automatically better

It's tempting to conclude that more engines means better results. We tested that. It doesn't.

We tried eight open-source Chinese engines running locally and let them vote. The result was worse than the best single engine on its own. Most of the eight came from the same company and were trained on heavily overlapping data, so they tended to fail on the same words. Voting didn't correct those errors. It locked them in.

Switching to four engines built on four different technical approaches changed that. Between any two of them, only 32.6% to 45.8% of the errors fell in the same place, and together they clearly beat any one alone.

So when we choose supporting engines for a language, we don't only ask how accurate each one is. We ask whether it fails in different places from the main engine. Korean is an example. One engine looked fine on its own, but its worst mistakes were almost identical to the other two: all three got the same words wrong in the same way. We replaced it with Soniox, whose errors overlapped less.

One more thing made us cautious. When AssemblyAI released a new version, its published error rate for mixed-language audio dropped from 9.07% in the previous generation to 7.69%. We checked it word by word in six of our languages. Every one came out either worse or no better: the new version added words nobody said. An English sentence gained a "Bruh". An Italian "50,000 lire" became "50 million". By error rate, nearly all six looked like improvements, because one extra word barely moves that metric. The official documentation explained it: the new version uses a large-language-model-style decoder. We rolled back to an older version.

A vendor's scorecard measures the vendor's chosen conditions. It isn't your interview.

Where we still lose

Having made the case for multiple engines, here's what we got wrong.

The final transcript still had 10 errors, all everyday words. In one, the interviewee said "my wife" and the transcript said "my sister".

Given the same input three times, our comparison step recovered 0, 3 and 3 errors. That step has randomness in it too.

We tried changing the comparison rules and ran an A/B test. It didn't help, so we rolled it back. Digging further, the problem wasn't in the rules at all but one step earlier, in alignment. The correct word was sitting in some of the supporting transcripts, but because the section boundaries didn't line up, it never reached the comparison. If the right answer isn't in the input, no rule can pick it.

We haven't fully solved that yet. We're including it because it's the same lesson as everything above: multiple engines are only worth as much as the evidence you can actually bring to bear.