What Is Speaker Diarization?
Speaker diarization is the process of figuring out who spoke when in an audio recording, without necessarily knowing who anyone actually is. It doesn't identify \"this is Marvin talking.\" It identifies \"Speaker 1 talked from 0:04 to 0:19, then Speaker 2 talked from 0:19 to 0:41.\" Naming the speakers is a separate step, usually done by a human after the fact, or by matching a voiceprint against a known sample.
Definition
What diarization actually solves
Run a two-person interview through plain speech-to-text and you get a wall of text with no sense of who said what. Diarization segments that same audio by speaker turn, so a transcript can render as an actual back-and-forth, Speaker 1, Speaker 2, Speaker 1, instead of one undifferentiated block. It's a separate model from transcription itself: one system converts speech to words, another figures out which voice each stretch of audio belongs to, and the two get merged into a labeled transcript.
The metric
How diarization error rate actually gets measured
The standard accuracy metric is Diarization Error Rate, or DER: the percentage of total speech time that gets misattributed, calculated as (missed speech, plus false alarms, plus speaker confusion) divided by total speech duration. A DER of 10% means one-tenth of the audio's speech time was assigned to the wrong speaker, missed as silence, or flagged as speech that wasn't there. Lower is better, and even the best systems don't hit zero, because real conversation includes overlapping speech, interruptions, and voices that sound genuinely similar.
Real benchmark numbers
What accuracy looks like on real recordings, not marketing copy
Diarization accuracy is usually reported against a handful of standard research datasets, and the AMI Meeting Corpus, a large public collection of recorded multi-person meetings built by a European research consortium, is one of the most cited. Recent benchmarking work puts pyannote.audio 3.1, the open-source model most self-hosted pipelines are built on, at roughly 12-14% DER on AMI, with better results (around 9-11% DER) on the cleaner VoxConverse dataset, and worse results (17-19% DER) on DIHARD III, a dataset deliberately built from harder, messier real-world audio. The commercial PyannoteAI service has been benchmarked as low as 11.2% DER on multilingual test sets, with the open-source DiariZen model close behind at 13.3%.
The takeaway isn't a single number, it's that accuracy depends heavily on the audio itself. Clean two-person interview audio, recorded on separate mics, performs far better than a five-person panel recorded on one room mic with cross-talk.
Engine comparison
What's actually running under the hood at different providers
| Engine | Type | Reported accuracy | Speaker limit |
|---|---|---|---|
| pyannote.audio 3.1 | Open-source, self-hosted | ~11-19% DER depending on dataset | No hard limit |
| WhisperX (Whisper + pyannote) | Open-source pipeline | Inherits pyannote's diarization accuracy | No hard limit |
| PyannoteAI (commercial API) | Hosted API | ~11.2% DER on multilingual benchmarks | No hard limit |
| AWS Transcribe | Hosted API | ~94-96% overall word accuracy on clear audio | Up to 10 speakers |
| Google Speech-to-Text (Chirp 3) | Hosted API | Meaningfully improved diarization vs prior versions | Varies by config |
| AssemblyAI | Hosted API | Competitive with major providers on clean audio | Up to 30 speakers |
None of these numbers are directly comparable across sources, since every provider benchmarks against different test sets. Treat the table as a map of what exists, not a ranked leaderboard.
Where it still breaks
The three conditions that reliably raise the error rate
- Overlapping speech. Two people talking at once is where nearly every diarization system loses the most accuracy. It's counted as both missed speech and speaker confusion in the DER formula.
- Similar-sounding voices. Two speakers of similar pitch and cadence, especially on a shared microphone, are harder to separate than voices with obvious tonal contrast.
- Compressed or phone-quality audio. Diarization models rely on subtle acoustic features to tell voices apart. Heavy compression, phone codecs, and background noise all strip out exactly the detail the model needs.
A concrete example
What this looks like on a three-person podcast recording
Take a typical three-person podcast: a host and two guests, recorded on a single shared room mic rather than individual lav mics. On clean, individually-miked audio, a modern diarization model will typically separate the three voices with very few errors, since each speaker's audio channel is isolated from the start. On the single shared mic, the same recording will show far more speaker-confusion errors around cross-talk and quick back-and-forth exchanges, and any moment where two people laugh or respond at once tends to get mislabeled or merged. The fix isn't a better model, it's better source audio: individual mics per speaker cut diarization errors more than any algorithm change will.
Why editors care
What diarization actually enables downstream
Accurate speaker labels are what makes role-based editing possible: filtering a transcript to only the interviewer's questions, or only one guest's answers, searching a specific speaker's lines instead of scanning the full transcript by eye. Get the diarization wrong and every downstream step, selects, cleanup, exports labeled by speaker, inherits that error. It's also why individually-miked audio is worth the extra setup time on any multi-speaker shoot: it's the single biggest lever on diarization accuracy, well ahead of which engine you're using.
Related reading: what a multicam edit is, what a two-column script is, transcript editing for journalists, and what text-based editing means.
Sources
"frequently asked questions
The process of determining who spoke when in an audio recording, segmenting the audio by speaker turn, without necessarily identifying who each speaker actually is by name.
With Diarization Error Rate (DER): the percentage of speech time misattributed, calculated as missed speech plus false alarms plus speaker confusion, divided by total speech duration. Lower is better.
The best-performing systems report roughly 11-19% DER depending on the benchmark dataset and audio quality, with commercial services like PyannoteAI reporting figures as low as 11.2% DER on multilingual test sets.
Overlapping speech, similar-sounding voices, and compressed or phone-quality audio are the three conditions that most reliably raise the error rate.
Significantly. Individual microphones per speaker cut diarization errors far more than switching between engines does, especially compared to a single shared room mic with cross-talk.
Pyannote is the diarization model itself, figuring out who spoke when. WhisperX is a pipeline that combines Whisper for transcription with pyannote for diarization into one workflow.
You might also like...
