Vietnamese: the 'hallucination' was a market vendor

At the end of a market scene in our Vietnamese tests, Doubao wrote a line the subtitles don't have: "Ai mua gà kìa", roughly "who's buying chicken?". The first report logged it as a hallucination. On review it turned out to be a real vendor calling out in the background. The subtitles left it out, and Doubao heard it.

There isn't much evidence for this language, just two valid clips. Here is what they do and don't tell us.

The test clips

Two Vietnamese films, scored against the Vietnamese subtitles for the deaf and hard of hearing from Netflix:

• Furie (2019): debt collection at a market pawn stall, then a pickpocketing accusation. Several people trading insults fast in a southern dialect, market noise, overlapping voices. Top level on our three-level difficulty scale.
• Song Lang (2018): backstage at a traditional opera troupe, fans dropping by to chat. Quick back-and-forth with crowd noise. Middle level, and much clearer than the first.

We wanted two more clips with denser conversation. Every YouTube "full movie" we found was either a re-cut by some other channel that didn't match the subtitles or blocked behind age verification, and the studios' own channels only carry trailers. We stopped at two rather than pad the set with audio we couldn't verify.

Engines we tried

ElevenLabs Scribe v2, AssemblyAI, Doubao and Speechmatics in the first round. Soniox was evaluated during a later review. Gemini was tested separately.

Where the engines crossed red lines

Dropped passages. AssemblyAI did this on both clips. It skipped the first four sentences of Song Lang entirely, which is the main reason it came last on the clear clip.

Substitutions that turn into names. AssemblyAI wrote dị ứng ("allergy") as Dĩ Ấn, which reads like a person's name, three times. In Furie it turned the name Mai into công an ("police"), ăn cắp ("steal") into Anh Cất, and tang chứng ("evidence") into ăn trứng ("eat eggs"). The whole scene is about whether someone stole something, and with those errors the story changes.

Invented sentences. Our primary engine, ElevenLabs, made up two names that don't exist at the loudest, most overlapped point near the end of Furie, along with a sentence, Họ đang buôn bán ("they're doing business"). Speechmatics opened Song Lang with a ca sĩ ("singer") nobody said.

Soniox had zero red-line errors of any kind on either clip.

What we chose

Primary: ElevenLabs Scribe v2. Top tier on both clips, and the most complete through the beginning and middle.

References, in order: Doubao, Soniox, Gemini. All three run on every Vietnamese file.

• Doubao: tied with the primary for first on both clips and closest to the subtitles. Its one "hallucination" didn't survive review.
• Soniox: replaced AssemblyAI. No red-line errors, 9.1% on the clear clip, and it recovered the most content on the noisy one.
• Gemini: our Vietnamese film clips are all arguments and crowd scenes, which say little about normal conversation. So we ran five minutes of a Vietnamese podcast through Gemini and the primary and checked them against each other. Both produced exactly 1,018 words. Gemini made one real error: chôn ("bury") became trốn ("hide").

Evidence against our choice

• The primary engine invented content at the end of Furie. In Vietnamese you can't assume the primary doesn't make things up; the noisy tail of a recording needs the reference tracks to check it.
• When Gemini was tested only on these two film clips, the verdict was "don't add". It was reversed after the podcast test, and that podcast had no reference transcript, so Gemini and the primary were only checked against each other.
• One clip is extremely noisy and one is fairly clear. We have nothing in between.

Open problems

We rate our confidence in this language's setup as medium, for a simple reason: two clips can't support a claim that anything is systematic. The subtitles for two more clips are ready; what's missing is licensed audio that lines up with them. Once we have it, the same method can take the set to four.