Italian: when our main track heard dismay as praise

In I soliti ignoti, a gang of would-be thieves has just watched some secretly shot footage, and one of them says, frustrated, "é finita la pellicola": the film's run out. Our main track, ElevenLabs, wrote "il film è bellissimo," "the film is gorgeous." Dismay turned into praise.

Of the ten engines that ran on this clip, only Doubao got that line right. AssemblyAI heard "benzina" (petrol), Gladia heard "bestia" (beast), and Speechmatics dropped it. Italian is our clearest example of a reference track correcting the main track.

The test clips

Four film scenes covering four regional accents:

• I soliti ignoti (1958, Rome): a gang watching surveillance footage, sizing up a safe, then haggling with a locksmith, with lots of overlapping speech.
• Baarìa (2009, Sicily): a crowded village-square scene with a barber, a post office and a borrowed copy of Marx.
• Ammore e malavita (2017, Naples): condolences at a funeral, then a widow putting a price on revenge.
• Il Capitale Umano (2013, Milan/north): a theatre board debating its program, in clear standard Italian.

On the three tiers from the first post, Il Capitale Umano sits near the middle tier and the other three are in the hardest. The basic tier isn't covered.

Several of the films have a score, musical numbers or background songs, so we cut around the passages where music drowns out the dialogue.

The candidates

ElevenLabs, AssemblyAI, Doubao, Speechmatics (two models), Soniox, Gladia, Deepgram, FunASR, Qwen, and later Gemini.

The six red lines, engine by engine

Dropped passages. Deepgram fell apart on all three hard clips. I soliti ignoti has 324 words in the reference; Deepgram returned 134. Baarìa has 121; it returned 64. The Naples scene has 103; it returned 15, losing the first 90 seconds of condolences and accusations entirely.

Flipped meaning. Besides the main track's line, Gladia turned "è stato Ciro che ha venduto Vincenzo" (it was Ciro who sold out Vincenzo) into a line about being betrayed by "vincenza," reversing who betrayed whom. Speechmatics turned "darà un milione" (will offer a million) into "Non era 1.000.000," adding a negation.

Invented words and sentences. Speechmatics turned the crime boss's surname, Strozzalone, into "stronza," an insult aimed at women. Gladia turned "ho fatto un regalo" (I gave a present) into a swear word. Soniox inserted a whole sentence in the Naples scene, "Il ciccio di Masellini," that nobody says.

Repetition loops. Qwen fell into a loop, repeating itself ten times at one spot, and also dropped the negation word mica, flipping a meaning.

Language drift. See the Gemini section below.

The Milan scene is the control case. On clear standard Italian, all seven engines were clean, and even Deepgram held together. Every one of the damaging errors above came from overlapping speech or heavy dialect.

What we chose

Main track: ElevenLabs Scribe v2. Clean on three of four clips, and far ahead of everyone on the Naples dialect. The flip in I soliti ignoti is its only red line in this language.

Reference tracks, in priority order:

1. AssemblyAI (universal-2): no invented sentences, no flips and no collapses across all four clips and all four accents.
2. Doubao: likewise no invented sentences, no flips and no collapses. It was the cleanest on the Rome and Sicily clips, and the only engine to get the pellicola line right.
3. Gemini: added on August 26, 2026.

AssemblyAI and Doubao were the only two engines in this round that never invented, never flipped a meaning and never collapsed on any clip.

Evidence against our choice

• The main track flipped a key line. This is the most direct evidence against our setup in Italian. We rely on the reference tracks to catch it, which only works if a reference track happens to be right at that spot.
• Our evidence for Gemini in Italian is thin. In the 26-language test, Italian was classed as "needs more testing": we had only one conversational clip, and Gemini was clean on it. On the two dialect-heavy crowd scenes, it came last both times. In Baarìa, where the actual line is "Dad, did I get invited?", it wrote an unrelated sentence roughly meaning "What a lovely ball, so many kids, I want it!" In Ammore, it produced a passage of English out of nowhere: "Don't leave us alone, don't be shy. When I was a child, I used to make." Gemini is still on the reference tracks, but in this language the evidence for it comes down to that one conversational clip.
• Speechmatics' other model is less harmful, but not harmless. Melia-1 didn't repeat the slur or add the negation, but it did repeat "un bianco" four times.

Open questions and limits

• All four clips are from films, the oldest from 1958. The audio is a long way from a modern interview recording.
• No basic-tier clip and no real interview. Gemini in particular still lacks a solid conversational sample in Italian.
• Names weren't scored. Engines routinely get them wrong, and the glossary handles that.