When interviews switch languages

Cross-border research produces a lot of recordings like this: the interviewee speaks Spanish, the person asking questions speaks Chinese, and an interpreter sits between them. The conversation goes back and forth between two languages.

We thought this would be straightforward. We read every engine's documentation, ran a round of lab tests and reached a set of conclusions. Then we tested a real recording and the conclusions flipped.

Two kinds of mixing, very different results

First, two situations need to be kept apart:

• The occasional aside: one main language, with the odd word or sentence in another, under a tenth of the recording.
• Taking turns: for example, consecutive interpreting, where the two languages alternate and each takes up roughly half.

Engines behave very differently in these two cases. They shouldn't be lumped together.

Some good news first: mixing in English is mostly fine. Nearly every engine's multilingual support is built around "language X plus English". Our main engine writes out a full English sentence as spoken, even in the middle of Spanish.

The trouble starts when neither language is English, such as Spanish mixed with Chinese.

The same "language" setting means four different things

Most engines have a language parameter. You might assume it does roughly the same thing everywhere. We went through each vendor's documentation and then tested each one. It's four different things:

• ElevenLabs: tells it which language the recording is mainly in.
• Soniox: a hint it will lean towards, without stopping it recognising other languages.
• Doubao: picks which language model to run. Oddly, leaving it blank is the worst option: it then listens for Chinese and English and forces the Spanish into English.
• AssemblyAI: decides what language the whole file is in, then hands the entire file to that one language's model. Of the two languages it can handle together, one has to be English.

That last one has a blunt consequence. In a recording that was 45% Chinese, AssemblyAI still decided it was Spanish (confidence 0.84). We tried four configurations. Not a single Chinese character came out.

There was a surprise in the other direction too. Reading Doubao's documentation, setting the language to Spanish should lock the whole file into Spanish. In practice it didn't. It still transcribed the Chinese in the middle.

The worst outcome is translation

With the occasional aside, our main engine fails in one of two ways, depending on how you set it up. (This was tested on spliced recordings. The next section covers the problem with splicing, but if the engine gets it wrong when the language change is that obvious, a real recording won't be any easier.)

• Tell it the main language is Spanish, and the Chinese sentences simply vanish. The timeline skips over them and leaves no trace.
• Don't tell it, and the Chinese sentences are translated into fluent Spanish. "你是谁?" became "¿Quién eres?". Not one Chinese character in the whole transcript.

The first leaves a gap. The second adds Spanish that nobody said. The second is worse, because the transcript reads flawlessly and a researcher would assume the interviewee actually said that in Spanish.

Another engine, one that uses a large language model to transcribe, got stuck at a language switch and repeated the same word about 370 times.

Real audio overturned the lab results

To test the taking-turns case, we started with spliced audio: existing Spanish clips and Chinese clips cut together. The reference transcripts came for free and it cost almost nothing.

The spliced tests said:

• Tell the main engine the main language is Spanish, and everything breaks: the Chinese gets translated, invented content appears, and the timeline falls apart.
• Leave it to work out the language itself, and it gets everything right.

Then we got a real interpreted interview: 66 minutes, Spanish and Chinese, with a researcher, an interpreter and an interviewee. The results were the exact opposite:

• Telling it the main language is Spanish gave a near-perfect transcript. Both languages switched correctly, and all three speakers were kept apart. Even the interpreter, moving back and forth between languages, was correctly recognised as one person.
• Leaving it to decide, it concluded the whole thing was Chinese (confidence 0.86), transcribed with Chinese as the main thread, and dropped whole passages of Spanish.

Why? Spliced audio has an obvious seam between languages: the room, the volume and the background noise all change at once, so it's easy for an engine to notice a switch. A real recording is one room and one microphone. There's no seam.

The rhythm was different from our splices too. Chinese made up 29%, alternating as "long Spanish answers of one to four minutes, short Chinese interjections of fifteen seconds to a minute or so", which fell right in the range our splices hadn't covered.

What we took away

How well a mixed-language interview gets transcribed depends on three things: which two languages, whether it's an occasional aside or taking turns, and how the engine is configured. The vendor documentation can't answer those questions, and neither can spliced test audio.

Only real recordings can.