Russian: the line both reference engines got wrong

Russian was where we first caught two reference engines making the identical mistake on the same line while the primary engine got it right. If we had settled that line by majority vote, the transcript would have been wrong.

Here is how we tested Russian and what that episode changed.

The test clips

Four scenes from Russian films, scored against the films' Russian subtitles for the deaf and hard of hearing. We picked all four for difficulty, so on our three-level scale they all sit at the top:

• Brother (1997): a man shouting through a door, several people cutting in outside, market noise underneath.
• Brother 2 (2000): fast phone talk packed with names, places and numbers, with a radio playing in the background.
• Leviathan (2014): a handful of men plotting around a table, lines overlapping.
• Kokoko (2012): a room full of women working and chatting, talking over each other.

We don't publish the audio. None of these is an ordinary one-on-one conversation at a normal pace, which comes back later as a limitation.

Engines we tried

Eight: ElevenLabs Scribe v2, AssemblyAI, Speechmatics, Doubao, Soniox and Deepgram, plus FunASR and Qwen as controls. Gemini was tested separately afterwards.

Where each engine crossed a red line

We judge engines mainly on six kinds of failure: invented sentences, flipped negations, repetition loops, inserted words, dropped passages and drifting into another language. In Russian we saw these:

Inserted words. In the Brother scene, Speechmatics added a term of address, Доченька ("little daughter"), where nobody says it, and turned the swear verb охерел into the name Кирилл. On review we counted at least eight inventions like this across its Russian output, some in every clip. Deepgram invents too: in "you couldn't get through him with a cannon (пушки)" it heard a town name, Пушкино.

A flipped negation. In Leviathan someone says «сегодня не в нашу пользу всё решится» ("today it won't be settled in our favour"). AssemblyAI dropped the не and wrote «сегодня они в нашу пользу все решатся», which says the opposite. The same passage has four or five other negatives and it got every one of them right, so we treat this as a one-off rather than a pattern.

A replaced sentence. «Двух жён в могилу свёл» ("he drove two wives to the grave") came out of Soniox as «Да уже он в могилу свёл», with a different subject. At first this was logged against Soniox alone. On review, five of the eight engines got that line wrong. The audio is simply hard there.

Language drift. Doubao's first Russian run produced a jumble of English, Japanese place names and Chinese. That one was our fault: the adapter wasn't sending a Russian language code, so the engine silently fell back to Chinese. With ru-RU set, all four clips came back as coherent Russian.

Repetition loops: AssemblyAI didn't produce any across the four clips.

What we chose

Primary: ElevenLabs Scribe v2. Lowest WER on three of the four clips. It lost first place on Brother 2 mainly because it transcribed the background radio, both the announcer and the song lyrics, which the subtitles leave out, so all of it counted as errors. Its Russian mistakes were mostly dropped words, such as a missing отсюда ("out of here"). We didn't see it invent a sentence.

References, in order: AssemblyAI, Doubao, Gemini. All three run on every Russian file.

• AssemblyAI: no loops and no runs of invented text in four clips, and the lowest score on Brother 2, the clip densest with names.
• Doubao: once the language code was right, no garbage and no inserted words; when it's wrong, it's wrong about one word, like hearing торге ("haggling") as твороге ("cottage cheese"). It comes from ByteDance and a very different technical lineage from AssemblyAI, so the two tend to fail in different places.
• Gemini: our Russian film clips are all shouting and crowd scenes, which tell you little about normal conversation. So we took five minutes of a real TV interview (Vladimir Pozner with Edvard Radzinsky) and checked Gemini line by line against the primary. No invented text, no flipped meaning, no loops. It got all four direct quotations and every proper name right, including Cambridge, Oxford and Nicholas II, and skipped three "of course"-type backchannels.

Evidence against our choice

• AssemblyAI isn't clean in Russian. On review it had three dangerous errors, including the negation above and a full swap of «Кто? Зинка я, открывай» ("Who's there? It's Zinka, open up") for «Куда ты отдаёшь, Зинка».
• The two-engines-agree-and-both-wrong case is exactly why three-way voting can't be trusted. Engine errors aren't independent, and on negation and polarity the side with more votes isn't necessarily right.
• On Brother 2, Gemini wrote four sentences of background radio into the transcript and turned "do you have a ten-day pass?" into "I have a ten-day pass". It scored worst of the four on that clip, 43.6%. As noted above, the primary picked up the radio too on another run. In a real interview, that's the equivalent of transcribing the TV in the meeting room as if the participant had said it.
• Soniox scored 12.3% on Leviathan, lower than the primary, and we still passed on it because review found whole-sentence replacements. Go by the score alone and you'd reach the opposite decision.

Open problems

• Our reviewers can't read Russian closely enough to judge every line. The first pass only checked what can be judged objectively: coherent Russian or not, garbage characters, inserted words, swapped sentences. The flipped negation turned up later, when we compared every negative word one by one. There may be meaning errors we still haven't found.
• All four clips are hard film scenes. We have no scored clip of one or two people talking at a normal pace. The Gemini interview had no reference transcript, so it was only checked against the primary.
• At the time of testing there was no dedicated safeguard against background audio ending up in the transcript.