Who said that? Speaker attribution when many people talk

A seven-person discussion, 37 minutes long. We gave the whole recording to ElevenLabs and asked it to label who said each sentence.

It found nine people.

The bigger problem was that people's labels kept changing. One speaker was labelled A from minute 8 to minute 18, became B just after 18:30, went back to A around minute 30, and turned into B again at 32:47. Across the seven speakers, identities switched 16 times. Measured by words, only 68.6% of the transcript was attributed to the right person.

For interview research that's worse than a few misheard words. You can't trust who said anything.

With more people, built-in speaker labels drift

That recording comes from AISHELL-4, a public dataset of Chinese meeting recordings, each with five to seven people around a table. We tested ElevenLabs' built-in speaker labelling on it repeatedly:

• On 10- to 15-minute clips, depending on the settings, between 67.5% and 97.8% of the content was attributed correctly, and one person's identity switched as many as 12 times.
• More people made it worse. On seven-person clips, even when told there were seven speakers, it managed only 77% to 82%.
• It couldn't reliably count the people. Left to work it out itself, it got the number right 4 times out of 12, and every miss was one person short: the quietest one or two speakers got merged into someone else.

Running the same settings twice could give very different results. One clip scored 81.9% on one run and 93.8% on the next.

Short clips flatter the results

Those numbers hide a trap of their own: the shorter the clip, the better it looks.

In the same meeting room, 15-minute clips scored 78% to 93%. The full 37-minute recording dropped to 68.6%, 10 to 25 points lower, and the failure changed from "one person short" to "extra people and identities swapping back and forth".

We've seen the same thing elsewhere. One open-source approach scored 99.8% on speaker identification over a 10-minute clip. Run on the full recording, it fell to 63.9%.

So we now judge only on complete recordings.

The scoring method can mislead too

There's another trap in the ruler itself.

Human reference transcripts usually time-stamp to the nearest second, and by convention several lines can share the same second (one person finishes and another jumps straight in). In one reference we used, 55% of lines shared a time-stamp with a neighbouring line.

Score by the timeline and a one-second offset counts as a whole sentence attributed to the wrong person. The same output scored 81% by timeline and 99.7% by text alignment.

A few counter-intuitive findings

• Allowing for more speakers doesn't find more speakers. We tried this with an open-source system: give it room for one extra person and, instead of finding a third speaker, it splits the most talkative person in two.
• A different model isn't necessarily different. Five voice models from the same developer produced word-for-word identical output. Only a model from another developer changed anything.
• Default settings erase "mm" and "uh-huh". Acknowledgements shorter than half a second were being thrown away as noise. Once we stopped that, more speaker changes were caught, and more accurately.

Our approach: learn each voice first, then attribute sentence by sentence

What we ended up with has three steps:

1. A "voice card" for each person: 10 to 30 seconds of clean speech from that person, like a round of introductions at the start. Below 10 seconds it gets noticeably worse; beyond 30 seconds it stops improving.
2. Attribute each sentence: ElevenLabs only transcribes and splits the sentences. Who said each one is decided by comparing that sentence's voice against every card, not by ElevenLabs' own speaker labels. Those labels aren't consistent over time: attributing by label got only 71.5%.
3. Sort by confidence: confident calls are made directly; uncertain ones are flagged for a person to confirm. Across two test meetings, the confident group covered 564 segments without a single error.

Very short "yeah"s and "mm"s don't carry enough voice to identify on their own, so they take the identity of the same person's longer sentences around them.

Results: above 99% on public meeting data, with no drift

Two complete AISHELL-4 meetings:

Both got the number of people right and had zero identity switches.

We also tested six 10- to 15-minute clips and a public five-person panel discussion (99.75%). All nine were above 97%.

Almost every remaining error is a "yeah", "right" or "okay" shorter than a second.

The limit: people talking over each other

That's public data. Real focus groups have been harder.

• A five-person open discussion: the moderator raises a topic and four participants jump in freely. All five are women, and about a fifth of the talking time has people speaking at once. The confident calls were about 97% right, but 30 to 40% of segments could only be flagged as uncertain.
• An eight-person focus group recorded at telephone quality: once every participant had a proper voice card, confident calls were 98.1% right, but uncertain material still made up 17.5% of the text. Our own bar is "confident calls above 99%, uncertain material at most 5%". This one didn't pass.

Listening back to the mistakes made one thing clear: when several people talk at once, human ears can't separate them either. Professional note-takers in the room rely on video.

So our commitment on speaker attribution covers multi-person discussions where people take turns: one person speaks, the others listen. Within that, we keep every person's identity consistent from start to finish. People talking over each other is outside it.