Finnish: near-perfect on clear speech, invented lines under gunfire

We tested two clear everyday Finnish recordings: a phone call to arrange a time, and a talk about a café that ends with someone ordering. ElevenLabs scored 2.7% and 2.9% WER, and reading them line by line, we found no real errors.

Then came the assault scene in the war film Unknown Soldier. Under the machine-gun fire and explosions, every engine started inventing sentences, ElevenLabs included.

The test clips

Four clips:

Complete original audio for Finnish films is hard to come by. The last two clips come from learners' podcasts where the presenters typed out word-for-word subtitles themselves, which has been a reliable source of reference transcripts for smaller languages. We deliberately avoided scenes in heavy regional dialects, such as those from Lapland or Karelia.

Engines we tested

ElevenLabs Scribe v2, Speechmatics, AssemblyAI, Soniox and Doubao, with Gemini added in August. When AssemblyAI released a new model, we tried that on Finnish as well.

Doubao doesn't support Finnish; it returned English gibberish.

The red lines

Hallucinated sentences, almost all in the assault scene. An officer yells "Boys, damn it, forward!" and ElevenLabs wrote something like "you've been beaten alive forward". For "third and fourth, left", ElevenLabs wrote "back to the position" and AssemblyAI wrote "Paulus predicts, left". Speechmatics produced "the request of the rabbits" out of nowhere, plus a run of sentences at the end that aren't in the film.

There was only one on the clear clips: in the colonel scene, AssemblyAI turned the rank "Lieutenant Colonel" into a string of meaningless syllables.

Swapped pronouns: in the assault, AssemblyAI turned "get the men moving" into "get us moving".

Dropped passages: in the same clip, AssemblyAI lost the entire opening.

Loops: in the assault scene, Gemini wrote "Vihollinen!" (Enemy!) more than 75 times, swallowing the whole ending, and invented two sentences at the start.

Language drift: on a clear clip, Soniox wrote an English sentence, "Drivers to load not there", and repeated itself. Doubao was English throughout.

What we chose

• Primary engine: ElevenLabs Scribe v2. Best or joint best on all four clips, near-perfect on clear speech, and not a single flip.
• Reference engines: Speechmatics, AssemblyAI, Gemini.

Speechmatics is first. No flips across four clips, and lower WER than AssemblyAI on all three clear clips (32.7% vs 35.6% on the colonel scene, 9.4% vs 12.8% on the phone call, 4.6% vs 7.5% in the café). Its mistakes on clear speech are small smudges, like capuccino missing a letter. On clear speech it doesn't make sentences up.

AssemblyAI is second. Apart from that military rank, it had no hallucinations, flips or loops on the clear clips. Its trouble is concentrated in sustained gunfire-and-shouting.

Gemini: no real errors, word for word, on the phone call.

We don't use Soniox for Finnish, the only one of these Nordic and Central European languages where we don't. An English sentence appearing in the middle of a clear Finnish recording is harder to guard against than anything AssemblyAI did, so Soniox is only a backup here.

Evidence against our choices

• Gemini's 75-fold loop is a real production risk: when it happens, a whole section of transcript is gone. We run it because it made no real errors on either everyday conversation.
• AssemblyAI's new model did worse than the old one on three of the four Finnish clips, including new hallucinations. We didn't switch, and the rollout was later reversed for all languages.
• ElevenLabs invented two sentences in the assault scene too. Our primary engine isn't safe in extreme noise.

What's still open

• The two clear clips come from podcasts made for learners, recorded in a studio. That's cleaner than most interview recordings.
• We haven't tested heavy dialects.
• Two of the four clips come from the same film.