NVIDIA's open speech model tops Hugging Face's ASR board by 0.07 points, but trails a 1.7B model on private sets

NVIDIA released Nemotron 3.5 Transcribe on October 28. It is an open-weight speech model with about 30B parameters, of which about 4B are active per token: a speech encoder feeding a hybrid Mamba-Transformer mixture-of-experts language model. One checkpoint transcribes 40 languages, translates 31, labels up to eight speakers in 13 languages, and answers questions about audio up to an hour long. The weights are on Hugging Face under the OpenMDW-1.1 license. NVIDIA's headline claim is that the model is "first among open-weight models" on Hugging Face's Open ASR Leaderboard and tied for first on Artificial Analysis' speech-to-text ranking.

We checked both leaderboards. The rankings hold, but the margins are tiny, and the leaderboard's private test sets point the other way.

Hugging Face: first by 0.07 points

On the Open ASR Leaderboard's English track, which the maintainers updated on October 28 to add the model, Nemotron 3.5 Transcribe averages 5.33% word error rate. That is ninth overall. The eight entries above it come from Otter.ai, Zoom, ElevenLabs, Microsoft and four others, and none of them has public weights. Seven have no Hugging Face repository, and the eighth, p99lab's audrey-0.7, has only a README and a "proprietary" license. So "first open-weight" is accurate.

The next open model, Apache-2.0-licensed Qwen3-ASR-1.7B, scores 5.40%. It has roughly one eighteenth of Nemotron's total parameters, and the leaderboard measures it at 819 RTFx, a throughput measure, against Nemotron's 462. That makes it about 1.8 times faster on the leaderboard's hardware.

NVIDIA's blog describes the leaderboard as measuring word error rate "on public test sets". The default average actually covers 11 columns: nine public datasets plus two private aggregates, "Private (scripted)" and "Private (conversational)". The private sets come from Appen, DataoceanAI and Voice Arena. The maintainers say they exist "to address risks of overfitting to public datasets".

Splitting the average shows where the 0.07 points come from:

The public averages are our own calculation from the per-dataset numbers on the leaderboard. Nemotron wins seven of the nine public sets, with the largest gaps on SPGISpeech (1.91% vs 2.57%) and VoxPopuli (1.89% vs 2.85%). Qwen wins the noisy-speech set URGENT24 and Voice Arena's conversational Monsoon split, as well as both private aggregates. On scripted private speech, Qwen makes about 20% fewer errors. IBM's Granite Speech 5.0 2B, at 5.68% overall, also beats Nemotron on the private scripted set (3.16%).

Two private columns can't prove overfitting, and we don't know what is in NVIDIA's training mix. About the 463,062-hour Granary v2 release, the blog says: "These figures describe the dataset release, not Transcribe's full training mixture or its sampling weights." It does mean that on audio no one can train on, the 30B model has no measured edge over a 1.7B one in English.

Artificial Analysis: a tie that rounds in NVIDIA's favor

Artificial Analysis lists Nemotron 3.5 Transcribe and Mistral's Voxtral Small (24B, July 2025) as joint leaders among open-weight models at 2.8% AA-WER. Unrounded, the page's data has Voxtral Small at 2.77% and Nemotron's best run at 2.81%.

AA-WER weights three datasets. Half the weight goes to AA-AgentTalk, Artificial Analysis' own held-out set of voice-agent speech, and a quarter each to cleaned VoxPopuli and Earnings22. Nemotron is clearly better on AgentTalk (1.78% vs 2.27%), as NVIDIA says. Voxtral Small is better on the other two: 1.75% vs 1.99% on VoxPopuli and 4.78% vs 5.69% on earnings calls.

There is a second caveat. The configuration Artificial Analysis benchmarks for speed and price, DeepInfra's endpoint at $2 per 1,000 minutes, scores 3.42% AA-WER, not 2.81%. The page doesn't name the setup behind the better number. Anyone using the hosted model should expect the higher figure until that is explained.

What we couldn't check

• Speaker attribution, the model's most distinctive feature. NVIDIA cites first place on David AI's 21-language diarization leaderboard (18.26% diarization error rate) and on VoiceArena's 14-system Diarization-Bench (32.4% cpWER). In both charts, NVIDIA's own models hold the top two spots: on David AI the runner-up is Nemotron 3 Diarization at 18.38%. We did not verify these tables, and we have seen no independent tests yet.
• NVIDIA's own comparisons. The multilingual (4.62%) and long-form (10.98%) results come from the model card's comparison, which NVIDIA assembled.
• The medical fine-tuning case study. It looks carefully done: a speaker-level split with 2,764 held-out clips, held-out WER cut from 22.44% to 12.17%, and about 14 GPU-hours on one GB300. But it is a single vendor study on one benchmark.
• Quantized and local versions. NVIDIA says its quantized versions and local deployment through NeMo-Speech.cpp are "coming soon". Community quantizations already appear on Hugging Face, but we haven't tested them. NVIDIA also says the model is built for recorded audio, not live streaming.

The takeaway

Nemotron 3.5 Transcribe does what its rivals on these leaderboards don't: it transcribes, labels speakers inside the generation step, translates and answers questions, all from one open checkpoint. That is the real reason to look at it. As a plain English transcriber, the leaderboards NVIDIA cites show a near tie. It leads by 0.07 points on one board and trails by 0.04 on the other before rounding, and a much smaller, faster open model does better on the private test data.