Nvidia Nemotron 3 Diarization

NVIDIA Nemotron 3 Diarization: real-time speaker labels at a cent per audio hour Blaxel is joining Baseten . Read more

AI models

NVIDIA Nemotron 3 Diarization: real-time speaker labels at a cent per audio hour

NVIDIA Nemotron 3 Diarization answers who spoke when, 300ms–1s behind live, alongside any ASR.

Authors

Ansel Erol

Last updated September 23, 2026

Share

TL;DR Use NVIDIA Nemotron 3 Diarization from Baseten’s Model Library , with presets for prerecorded and streaming diarization, and real-time diarized transcription. This is an open, end-to-end diarization model that replaces the usual segmentation-plus-embedding pipeline and its tuning with a single pass, labeling up to eight speakers at a configurable interval as low as every 320ms. We optimized it on an RTX PRO 6000, which can sustain over 500 concurrent hour-long diarization streams (190 with transcription added). The same per-speaker activity can be used to inform VAD, endpointing, and turn-taking for voice agents.

An end-to-end streaming architecture Speaker diarization segments an audio stream by when each distinct speaker is talking, outputs timings for each distinct voice, and assigns every span a consistent speaker label. In conjunction with transcription, diarization can attribute transcribed words to who said them in a conversation. The NVIDIA Nemotron 3 Diarization model has a ~100M-parameter transformer-based architecture built on top of NVIDIA Streaming Sortformer . It turns 16 kHz audio into 10ms frames and emits a T×8 matrix of speaker-activity probabilities. Two pieces of state carry a conversation across chunks: an arrival-order speaker cache (speaker_0 is the first voice heard, and stays that way) and a FIFO (“First In, First Out”) of recent frames, with no embeddings or clustering required for an unbounded audio length. Latency configuration for diverse use cases While prior diarization models require extensive configuration of several different parameters or lack support for multiple latency profiles entirely, the same Nemotron 3 Diarization checkpoint can serve each request at four algorithmic latencies between 0.32 and 30.4 seconds, referring to how much future audio the model sees before it commits a label. To accommodate diverse application requirements ranging from real-time agent interactions to offline media processing, Nemotron 3 Diarization supports flexible algorithmic latency settings within a single unified checkpoint. ✕ Figure 1 — Latency profiles of a single checkpoint. Figure 1 illustrates how these latency profiles map directly to chunk sizes and right-context lookahead windows. Moving from the ultra-low profile to the standard low profile trades a modest increase in buffer delay for continuous quality gains. This predictability enables developers to configure buffer sizes dynamically per request without redeploying underlying model instances. High quality across four configurable latency profiles We report diarization error rate (DER) with overlap included and no collar. Nemotron 3 Diarization holds its accuracy as latency drops. Moving from the 30-second offline profile to the 0.32-second ultra-low profile increases DER by only 1-2 points compared to prerecorded, with quality sustained beyond four speakers. On AISHELL-4 at the “low” profile, we observed a DER of 9.8% against 27.2% for Streaming Sortformer v2.1, a significant improvement compared to this model’s predecessor. To evaluate standard diarization accuracy across varied acoustic environments and speaker counts, we measured DER across benchmark datasets without applying a time collar. ✕ Figure 2 — Diarization error rate by dataset and latency profile. The benchmarking results highlight consistent stability across multiple benchmark datasets in multiple languages, such as NOTSOFAR, AMI, CALLHOME, and AISHELL, outperforming Meta Muse Voice Transcribe on all datasets at even the ultra-low configuration and losing to pyannote community-1 only on AMI.  Even under ultra-low latency configurations, the error rate remains within a tight margin of offline baselines, confirming robust speaker attribution under real-time constraints. Streaming diarization on Baseten remains close to prerecorded quality across the tested latency profiles. Speaker activity as a voice-activity and turn-taking signal Speaker-attributed time segments are only one way to use Nemotron 3 Diarization. At its core, the model tracks speech activity per speaker in 10ms increments. This raw data can be used by voice applications to support three key capabilities: Voice activity detection : By combining data across all audio channels, it detects speech directly from the audio stream rather than relying on volume levels.

Speaker-specific end-of-turn detection : It tracks when a specific speaker stops talking for a set duration. This helps an AI voice agent distinguish between the primary user finishing a sentence and background chatter from someone else in the room.

Turn-taking and interruptions : The order in which voices appear identifies the primary speaker. If a second voice starts speaking, the model detects an interruption; if both speak at once, it flags overlapping speech, responding in roughly 300ms on ultra-low latency settings.

Day 0 support for prerecorded, streaming, and speaker-attributed transcription To streamline deployment across different audio infrastructure designs, Baseten exposes the model through three specialized serving presets. Figure 3 provides an architectural overview of these presets along with their target workloads and throughput characteristics. ✕ Figure 3 — Three presets on Baseten. These modular presets allow voice applications to choose between stateless batch requests, persistent WebSocket streams for live speaker tracking, or joint end-to-end transcription pipelines, matching backend infrastructure requirements directly. Each latency level runs as its own ready-to-use instance, so you can easily choose your speed per request without redeploying. The real-time transcription option pairs the speaker tracker with NVIDIA's 600M English speech recognition model, NVIDIA Parakeet V2 ASR , which is also in the Nemotron speech family. By using speaker activity to guide transcription on the same audio stream, it accurately captures both sides of a conversation, even when people talk over each other. Combining speech recognition with diarization typically requires managing separate model pipelines and aligning transcripts...