ASR Hears the Words, Not the Speaker

NVIDIA has released Nemotron 3 Diarization as an open-weight speaker diarization model on Hugging Face. Its task is specific: determine who is speaking at which point in a multi-party conversation. It can track up to eight speakers, handle simultaneous speech, and use one checkpoint for both offline recordings and real-time streaming. The model has 100 million parameters and runs on Linux through NVIDIA NeMo with Ampere, Ada Lovelace, Hopper, or Blackwell GPUs. Its weights are released under the OpenMDW License 1.1, which permits commercial use.

Diarization is often treated as a supporting feature of speech recognition, but the distinction matters. ASR turns sound into words; it does not reliably identify whether a sentence came from the moderator, a customer, or another engineer. Without attribution, a meeting summarizer cannot safely determine who made a commitment, call analytics cannot separate an objection from an agent response, and a voice agent cannot attach information to the right conversational source. Diarization produces anonymous speaker activity intervals, which can then be combined with ASR output to create a speaker-attributed transcript.

The Key Change Is How Overlap Is Represented

Compared with NVIDIA’s earlier Streaming Sortformer checkpoint, the most visible change is raising the supported speaker count from four to eight. The more important change is that the model does not assume only one person can be active at a time. It produces a [T, 8] tensor of per-speaker activity probabilities, allowing multiple channels to be active in the same frame. When people interrupt one another or speak over each other, the system does not have to force the audio into a single-speaker segmentation first.

That behavior follows from a concrete architectural choice. The model accepts 16 kHz mono audio, computes Mel-spectrogram features at a 10 ms step, stacks them by a factor of eight into 80 ms encoder frames, processes those frames with a 31-layer Transformer using rotary positional embeddings, and uses a Conv1D layer to upsample predictions back to 10 ms resolution. Speaker channels are assigned by arrival order and retained across streaming chunks rather than rematched from scratch for every chunk. The Arrival-Order Speaker Cache preserves information from earlier chunks, while a FIFO queue supplies recent frame context, together keeping anonymous labels stable over time.

Low Latency Is a Set of Trade-offs, Not One Number

The model card presents four operating points, showing why deployment cannot be summarized by a single latency number. The offline-style configuration has 30.4 seconds of input-buffer latency, a 12.73% DER on the full DIHARD III set, and 15,113× RTFx in batched throughput. The low-latency, very-low-latency, and ultra-low-latency settings use 1.04 seconds, 0.64 seconds, and 0.32 seconds of latency, with DER values of 13.18%, 13.28%, and 13.55%, and throughput of 865×, 579×, and 292× RTFx respectively.

For architecture decisions, the evidence is more useful as a trade-off curve. Reducing the buffer from 30.4 seconds to 1.04 seconds increases DER by only 0.45 percentage points, but lowers throughput from 15,113× to 865×. Pushing down to 0.32 seconds adds another 0.37 percentage points of DER and reduces throughput to 292×. NVIDIA recommends 0.32 seconds as the lowest setting, even though an 80 ms buffer is technically possible. These figures exclude model computation, networking, and ASR time, so they are not equivalent to end-to-end time from speech to a user-visible attributed transcript.

The Benchmark Shows Progress, With Clear Boundaries

The public evaluations suggest that Nemotron 3’s improvement is not only a consequence of supporting more speakers. In Voice Arena’s initial Diarization-Bench results, it ranked first among 12 systems and 17 configurations. The test covered 139 English conversations totaling roughly 22 hours. Its DER was 14.72%, compared with 19.3% for the next-ranked system, a relative reduction of about 24%. NVIDIA also notes that Voice Arena has not completed its Version 1 evaluation, so the ranking and numbers may change.

Against the four-speaker baseline at 1.04 seconds of latency, Nemotron 3 reduced DER in all eight evaluation conditions. Relative reductions ranged from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM, with an unweighted mean reduction of 41.0%. The result is not a universal sweep, however. On two-speaker CALLHOME at the 30.4-second setting, DER increased from 5.68% to 5.98%, while full-set CALLHOME-Part2 improved from 10.32% to 9.10%. Supporting eight speakers and being more accurate in every scenario are different claims.

For Product Teams, Anonymous Labels Remain a Boundary

The training mix helps explain why the model may handle more complicated multi-party audio. NVIDIA trained it on about 10,000 hours of real conversations and 82,611 hours of simulated multi-talker mixtures, including licensed real-world multi-speaker audio from David AI. Adding that data reduced compound DER from 11.19% to 10.42%. Those figures describe a change in the training recipe and aggregate error, not a substitute for validation in a target product. Language, microphone layout, call compression, and patterns of overlap can all change the outcome.

Deployment also requires separating diarization from identity recognition. Nemotron 3 emits anonymous channels assigned by arrival order; it does not know that “speaker 1” is a particular person. A product that needs stable customer, agent, or meeting-member identities must add downstream matching using account data, voiceprints, session metadata, or another mechanism, while accounting for label drift and false matches. The practical judgment is straightforward: if the bottleneck is overlapping speech, streaming chunks, and speaker attribution, this model deserves a place in the candidate architecture. If the requirement is inexpensive single-stream transcription or immediate real-name identification, open weights and benchmark performance are not a complete solution.