French: three engines, one sentence, the same wrong meaning

In the dinner-party argument in Le Prénom, someone says "même s'ils se trompent," meaning "even if they're wrong." Deepgram, Gladia and Soniox all wrote "même s'ils ne se trompent pas": "even if they're not wrong." Three unrelated engines, same spot, same flipped meaning.

Asking several engines to vote only works if their mistakes are independent. This one sentence shows that they aren't always.

The test clips

Four film scenes, each a different kind of hard-to-follow French:

• La Haine (1995): the Paris banlieue, street slang (verlan), heavy background noise. A TV crew gets heckled by a crowd, then a neighbor's car is smashed and everyone starts shouting.
• Le Prénom (2012): standard Parisian accent, a fast dinner-table debate full of names.
• Le Dîner de Cons (1998): standard Parisian accent, rapid comic back-and-forth, a phone farce built on a mixed-up name.
• Bon Cop, Bad Cop (2006, Canada): Quebec joual, including a full lesson in Quebec swearing.

The third slot was meant to be Bienvenue chez les Ch'tis, for the northern French dialect. The source we found had the right title and length, but a test run showed it was a compilation of director interviews and behind-the-scenes footage. We had to swap films, so the northern dialect went untested. The first source for the fourth clip was fake too: an English-language AI film review.

On the three tiers from the first post, all four clips sit between the middle and hardest tiers, with La Haine and Bon Cop the hardest. The basic tier isn't covered.

The candidates

ElevenLabs, AssemblyAI, Doubao, Speechmatics (two models), Soniox, Gladia, Deepgram, FunASR, and later Gemini.

The six red lines, engine by engine

Flipped negation. Besides the three-engine flip above, Speechmatics turned "c'est pas moi qui ai les tunes" (I'm not the one with the money) in La Haine into "c'est pas moi qui t'ai tué" (I'm not the one who killed you). Speechmatics' other model, Melia-1, also flipped the Le Prénom line.

Invented sentences. La Haine opens with no dialogue, and Gladia and Soniox each invented an opening line. Gladia rewrote "Est-ce que vous avez participé aux émeutes" (did you take part in the riots) into a garbled sentence with a negation that wasn't there. Speechmatics made up "niveau budget" at the end of Le Dîner de Cons. Deepgram turned San Francisco's "Golden Gate" into gibberish.

AssemblyAI invented things too. The Quebec oath "je m'en câlice" (I don't give a damn) became "je mets un cornichon" (I'm putting in a pickle). And in a film from 1995, it transcribed the words "YouTube" and "Covid," neither of which existed yet.

Repetition loops. Speechmatics rendered "Oh là là" as eighteen "la"s in a row. On its first run of Le Dîner de Cons, Gemini repeated "là" 1,659 times and never produced a transcript. A second run finished normally in 35 seconds.

Dropped passages. On La Haine, Deepgram returned 109 words where the others returned about 620; the second half, the fight over the smashed car, was simply gone. Doubao also dropped a lot on the Quebec clip, returning only 142 words.

Language drift. No large switches into another language on these four clips.

What we chose

Main track: ElevenLabs Scribe v2. No flipped meanings on any clip, it keeps verlan intact, and it was the second most robust on the Quebec clip.

Reference tracks, in priority order:

1. AssemblyAI (universal-2): no flips on any clip, and the cleanest of the group on verlan, debate and comedy.
2. Doubao: the only non-main engine that never invented and never flipped a line across all four clips. It degraded badly on the Quebec clip, but only by mishearing and dropping words.
3. Gemini: added on August 26, 2026. The four film clips are all crowd scenes and shouting, which told us nothing useful about it. So we took a five-minute stretch of a real long-form interview with Nicolas Sarkozy and compared Gemini with ElevenLabs line by line: no invented sentences, no flips, no loops, two minor changes in meaning, and one place where it was more accurate than ElevenLabs. The large omissions it showed on film clips didn't appear in the interview.

Evidence against our choice

• AssemblyAI isn't clean in French. The pickle and the two anachronisms are both its work. It had four invented passages across the four clips.
• Both reference tracks can fail at once. In one noisy scene, Doubao dropped the whole passage while AssemblyAI invented something in it, leaving the main track as the only usable witness.
• The main track invented things too. ElevenLabs had no flips, but it did have two invented passages, one of them at the start of a clip.
• We passed on the best engine for Quebec French. Speechmatics handled Bon Cop best. But it flipped a meaning in La Haine and invented lines elsewhere, so we left it out.
• Our evidence for Gemini in French is thin. One five-minute interview, with no reference transcript, checked only against the main track. On film clips it also has that 1,659-repetition failure on record, even if a rerun got through.

Open questions and limits

• The northern Ch'ti dialect went untested, and we have no clips for Belgian, Swiss or African French.
• All four gold clips are from films, and none is in the basic tier. The only interview is the one we added for Gemini.
• The three-engine flip is a reminder that disagreements near a negation can't be settled by counting votes. We're still adjusting how the merge step handles those spots.