Thai: part of our primary engine's score was our own bug

In our Thai tests, part of the primary engine's good score came from a bug in our own code.

One function in our pipeline strips punctuation from the ends of words. Thai vowel and tone marks are combining characters that sit on top of or below the letters, and that function treated them as punctuation and deleted them. So the stored ElevenLabs transcripts had lost every vowel and tone mark, and its error rate was calculated as if those marks didn't exist. The bug is fixed, but how accurate ElevenLabs really is on Thai diacritics still has to be spot-checked.

Keep that caveat in mind for every number below.

The test clips

Three clips, scored against hand-made Thai subtitles from the same source:

• A cooking interview: one or two people talking at a moderate pace with little background noise. This is our baseline clip, close to easy on our three-level difficulty scale.
• A reality show: several guests talking over each other and laughing, with background music and sound effects. We class it as middle level.
• A werewolf-style debate: people accusing each other, speaking fast, cutting each other off, mixing in English words like Seer and defend. Top level.

We had a fourth clip, but checking showed the audio and subtitles were about 30 seconds apart, which would have meant grading against the wrong answers, so we dropped it. Thai subtitles that are free, conversational and verbatim are rare, and three clips is the most we've been able to find.

Engines we tried

Ten: ElevenLabs Scribe v2, AssemblyAI, Speechmatics (two versions), Doubao, Soniox, Deepgram, Gladia, FunASR and Qwen. Gemini was tested separately.

Where the engines crossed red lines

Repetition loops. The debate opens with a burst of laughter that the subtitles record as one short laugh. After the laugh ฮ่า, Gladia wrote the Thai repetition mark ๆ more than 400 times in a row, across two lines, then cycled through the same two sentences four times. That's what pushed it to 83.9%. It never looped like that on the other two clips. AssemblyAI loops too: on review we found it had repeated one word for about 22 lines on the reality show and cycled another phrase about 25 times.

Dropped passages. On the debate, AssemblyAI skipped big stretches, in places close to ten seconds with no output. It came last on the first two clips, and on the debate only Gladia's collapse kept it off the bottom.

Inserted content (withdrawn). At one point we thought Qwen had invented a passage at the start of the debate. On review, the subtitles turned out to be about 10 seconds out of sync at the start, and Soniox, Doubao and FunASR had all transcribed the same content in the same place. Someone really said it. We withdrew that finding.

What we chose

Primary: ElevenLabs Scribe v2. No collapse and no runaway repeats on any clip; 12.6% on the cooking interview and the lowest score on the reality show. With the caveat above.

References, in order: Soniox, Doubao, Gemini. All three run on every Thai file.

• Soniox: the steadiest across all three clips, with the cleanest segmentation, following the subtitles line by line, with no loops and nothing large left out.
• Doubao: produces real Thai, lands level with Soniox and the primary on the debate, and leaves the loanword defend as it is. The report first said Doubao hadn't been run on the cooking interview. It had, and in places it was more accurate than Soniox there; the result just never made it into the table. It also comes from a different technical lineage than the first two.
• Gemini: on the debate it got one name wrong in the vote result, and all four engines finished within 1.9 points of each other.

What we left out: Qwen had the best score on the debate (18.3%), but it returned the whole three minutes as a single line with no word-level timestamps, which our alignment step can't use. FunASR's score was decent too, but its output is one unbroken block of characters with no sentence breaks. Gladia is out for looping and AssemblyAI for what it drops.

Evidence against our choice

• The biggest one is at the top: the primary's Thai score is inflated, and its real diacritic accuracy hasn't been spot-checked yet.
• Our reviewers don't read Thai. The report only judged what you can see without knowing the language: runaway repeats, sudden jumps in length, missing chunks, everything returned in one block. Whether any engine reversed or garbled the meaning of a line, we didn't check sentence by sentence.
• Gemini has only one conversation-style Thai test clip, which isn't enough. In an earlier round it also produced things like laughter tags and non-Thai labels; neither showed up again on this clip. Thai also once hit its output length limit.
• Soniox is itself a large-language-model-style transcriber. We generally worry that engines like this smooth over the parts they can't hear. We didn't see that in Thai, but we didn't check line by line either.

Open problems

• Thai doesn't mark word boundaries with spaces and we don't yet have a Thai word segmenter, so our alignment step works at the sentence level and is fairly coarse.
• None of the three clips is a real research interview.
• The primary engine's diacritic accuracy still needs checking.