Mandarin: choosing engines through gunfire and screaming

In a rescue scene from The Wandering Earth, an English disaster broadcast cuts through the Mandarin shouting. Speechmatics turned that broadcast into "Team Violence Eduardo Saverin Fear". Eduardo Saverin is a character from a different film, The Social Network. On the same passage, AssemblyAI translated the English into garbled Chinese, and FunASR returned nothing at all.

Chinese is the only language where we run four reference tracks. This post explains how we picked them.

The test clips

Four scenes from Chinese films, three to five minutes each, all in Mandarin:

• Let the Bullets Fly, the opening hold-up negotiation: clean recording, fast talk, people cutting each other off, lots of names.
• Assembly, a battlefield siege: gunfire over speech, sparse orders, wounded soldiers shouting on top of each other.
• The Wandering Earth, the Shanghai rescue: disaster-scene noise, with an English broadcast inside the Mandarin.
• Aftershock, the earthquake ruins: a mother screaming for her children, rumbling aftershocks, muddy audio.

On the three-tier scale from the first post in this series, Let the Bullets Fly sits near the middle tier and the other three are in the hardest tier. None of the clips covers the basic tier: one or two people talking calmly in a quiet room.

We loaded three of the four clips with heavy noise on purpose. The most dangerous error, an invented sentence, only shows up when speech is half-intelligible and buried in noise. On clean audio, none of the engines produces it, so you can't test for it there.

The candidates

Doubao, iFlytek, FunASR (flagship and flash versions), two Qwen models from Alibaba, ElevenLabs, AssemblyAI, Speechmatics, Soniox and Deepgram: eleven in all. We later ran the open-source Whisper large-v3 as a baseline.

The six red lines, engine by engine

Invented sentences. Speechmatics made things up on all three noisy clips. In Assembly it produced a sentence that roughly reads "I told you long ago… you want someone with you, I said with a smile." In Aftershock it produced "I love sports, I'm good, hello everyone," and turned the screaming into "baa baa baa baa." Whisper invented "President Li, early, early" in the shouting of The Wandering Earth, and rendered "There's a child here, someone come quick!" in Aftershock as "It's a bun, it's a bun, bun."

Repetition loops. Whisper looped on all four clips, even the clean one. At the end of Let the Bullets Fly, the word for "endure!" became a meaningless token followed by the same syllable thirteen times. Qwen's omni-plus model collapsed entirely on the screaming in Aftershock.

Flipped negation. Doubao, ElevenLabs, AssemblyAI and iFlytek produced no clear negation flips on any clip. Whisper came close once, turning "Heizi held on" into "Heizi stopped."

Dropped passages. AssemblyAI tends to stop when the noise gets heavy. On Assembly it transcribed roughly the first 30 characters and nothing after; on Aftershock it skipped the noisy opening. It didn't invent anything, but a reference track with a hole in it can't check anything either. Whisper was worse: the first 70 seconds of Aftershock came out as a run of two repeated characters, and the mother's cries for help disappeared.

Language drift. This is the English broadcast from the opening. On review, Doubao, ElevenLabs, iFlytek and Soniox all kept the English as English. FunASR and omni-plus dropped it, AssemblyAI translated it into Chinese, and Speechmatics turned it into a name.

Inserted content. ElevenLabs added one line in Assembly, roughly "the machine gunner captured the deputy regiment commander." None of the other ten engines produced it, and the reference is blank for that stretch. We can't tell whether it was really said, so we logged it as unverified.

What we chose

Main track: ElevenLabs Scribe v2. No negation flips on the three noisy clips and no confirmed invented sentences, apart from the one unverified line above. Speaker labels come from this track only.

Reference tracks, in priority order:

1. Doubao: no invented sentences and no flips on any of the four clips, the most complete output, and it kept the English broadcast intact.
2. FunASR: never invented a sentence on the three noisy clips. Its errors were omissions and mishearings.
3. iFlytek: among the most accurate engines when the audio is clean, with 12.2% CER on Let the Bullets Fly.
4. Gemini: added on August 25, 2026. It's an end-to-end large model, so it fails differently from the three acoustic engines. It returns whole sentences, not word-level timings, and it plays no part in deciding who is speaking.

All four run on every file.

Evidence against our choice

• FunASR is the weakest of the three acoustic tracks. It wrote "two hundred thousand" as "20.1 million." In Let the Bullets Fly it dropped two short answers that carry the scene, and where Mandarin and English mix it tends to lose the English entirely. We keep it because it holds up in noise and doesn't invent. Accuracy isn't the reason.
• The order of iFlytek and Doubao was never settled. The clean clip suggested iFlytek was at least as good as Doubao, with FunASR last. But that is one clean clip, and the re-check has been pending ever since.
• We passed on a more accurate engine to avoid invention. Qwen's omni-plus had the lowest CER on both Assembly and The Wandering Earth. It also invented a sentence in a faint passage at the end of The Wandering Earth and collapsed on Aftershock, so we left it out.
• The main track makes bad mistakes too. In Aftershock, ElevenLabs heard the name the mother keeps shouting, "Daqiang," as "big sister" throughout, and turned "the glass won't hold" into "I can't hold on."

Open questions and limits

• All four clips are from films. None is a real interview, and none is in the basic tier.
• ElevenLabs was never run on the clean Let the Bullets Fly clip. It held up on the three harder clips, but that cell is empty.
• We haven't checked Gemini line by line on these four public clips, so we have no Mandarin evidence for it to show you here.
• Names weren't scored. Errors like "Daqiang" are handled by the glossary, and this round of tests doesn't evaluate that.