Turkish: how one model upgrade broke a clean record

In Turkish, AssemblyAI started out with a clean record: four clips and not one line where it flipped the meaning. Then its maker shipped a new generation of the model and we briefly switched to it. All four Turkish clips got worse, some with close to double the error rate, and it turned the command indirin ("lower it") into indirmez ("he won't lower it"). That ended the clean record. We went back to the older model.

Same engine, different version, and the conclusion may no longer hold. That's the backdrop to how Turkish is set up.

The test clips

Four Turkish film scenes of about a minute and a half each, scored against Turkish subtitles for the deaf and hard of hearing:

• Vizontele (2001): a village headman welcoming visitors, with fast replies, interruptions and shouting.
• Düğün Dernek (2013): a camel has lost the bride; four or more people talk at once, mumbling and swearing.
• G.O.R.A. (2004): boarding a spaceship in a sci-fi comedy, orders piling up over dense sound effects and music.
• Bergen (2022): a birth mother turns up for a confrontation; rapid, emotional exchanges in the Adana dialect, with swearing.

The first two are top level on our three-level difficulty scale and the other two are middle level. There's no easy clip.

Engines we tried

Four in the first round: ElevenLabs Scribe v2, AssemblyAI, Speechmatics and Doubao. Soniox was evaluated in a later review. Gemini was tested separately.

Where the engines crossed red lines

Invented sentences. Speechmatics produced two or three in Vizontele. Hiç lüzüm yok ("there's no need") became İç sesim yok ("I have no inner voice"), and it tacked gazeteye getirip ("bring it to the newspaper") onto a sentence that doesn't say that.

An added negative. Same engine, same clip: Fikri benim adım ("my name is Fikri") came back with a yok ("not") on the end.

Dropped endings. On two hard clips Gemini's transcript is fine until the last stretch, then 20 to 25 words are missing, four whole sentences in one case. It ranked second by WER on both, so the score gives you no hint.

Language drift. Doubao's first Turkish runs came back as English gibberish, for example "I'm shaking these diseases". We hadn't sent a Turkish language code. With tr-TR set, all four clips came back as coherent Turkish.

AssemblyAI (on the older model) and Doubao had no invented sentences and no reversals across the four clips.

What we chose

Primary: ElevenLabs Scribe v2. First on every clip except G.O.R.A., where the sound effects flatten everyone.

References, in order: AssemblyAI, Doubao, Gemini. All three run on every Turkish file.

• AssemblyAI: the most accurate engine after the primary, with no invented sentences and no reversals. It does mishear names, for example Belgin as belli ("obvious"), but the other tracks can catch that kind of error.
• Doubao: last by WER on every clip, dropping to 28.6% on the dialect scene. Nearly all its mistakes are spelling. It writes dialect pronunciation out as it hears it, so pabuç becomes pafuç. But it never invented or reversed anything. For a reference track we cared more about whether an engine writes things that are confidently wrong than about where it ranks.
• Gemini: runs as the third reference.

Soniox did well in review: no red-line errors of any kind, and second among the non-primary engines. We didn't swap it in, because AssemblyAI is already the strongest non-primary engine and a switch had nothing to gain. It's first in line as a backup and would step in if Doubao starts dragging results down in production.

Evidence against our choice

• The upgrade story shows that AssemblyAI's "no reversals" is a record for one model version. A new version means testing again.
• Doubao is the weakest of the four. With heavy dialect and fast speech it adds limited information.
• Gemini has only one conversation-style Turkish test clip, which isn't enough to judge it. On G.O.R.A. its missing ending came with meaning changes as well: "take off your glasses" became "gouge out your eyes", and words like Ölmüş ("he's dead") appeared from nowhere.
• The report logs one minor invented phrase from the primary on each of the first two clips. But review also found these subtitles leave out a lot, and twice real speech had been misjudged as a hallucination for that reason. We can't be fully sure those two count against the primary.

Open problems

• All four clips are films. There's no real interview and no easy, everyday conversation.
• The subtitles leave out so much that when several engines agree on something the subtitles lack, the subtitles have to be rechecked before anyone calls it a hallucination.
• Gemini's missing endings don't show up in the score. Only a line-by-line read catches them.