NVIDIA Shows How to Fine-Tune Nemotron 3.5 ASR for Saudi Arabic Dialects

Why dialects trip up multilingual speech models

NVIDIA Nemotron 3.5 ASR supports multilingual streaming transcription across 40 language-locales, including Arabic. Even so, NVIDIA says regional dialects and local recording conditions are often underrepresented in pretraining data, so a model that scores well on broad benchmarks can still fall short in deployment.

Saudi Arabic is the example in the post. A model may handle Modern Standard Arabic or English yet struggle with Najdi and Hijazi speech. Fine-tuning only on the target dialect can improve it while weakening other languages, a problem known as catastrophic forgetting.

The workflow: curation, replay, and partial unfreezing

The recipe uses the NVIDIA NeMo framework and is built around a few techniques:

• Minimal curation: removes unusable labels and obvious alignment failures without discarding scarce, difficult speech.
• Replay mixing: interleaves a small amount of previously learned data (the post uses FLEURS) to reduce forgetting.
• Partial encoder unfreezing: limits how many parameters change, which is cheaper and faster than a full fine-tune, at some cost to accuracy.
• Length bucketing: reduces padding and makes training practical for streaming encoders.

The target data came from the SADA 2022 dataset, filtered to Najdi and Hijazi. Checks on clip duration, character rate and inaudible-speech markers kept 103,559 of 125,490 utterances, or 133.7 hours (82.5% of the starting set). NVIDIA stresses that the goal is to remove structurally bad examples, not hard accents or noisy speech. It also notes that a default UTMOS threshold of 3.0 would have rejected almost everything, so the team inspected score distributions and set corpus-specific cutoffs in NeMo Curator instead.

Results and trade-offs

According to NVIDIA, fine-tuning on the 133.7 hours cut word error rate (WER) from 55.05% to 29.96% on the target test split. English WER improved from 11.04% to 10.42%, and other Arabic dialects were not degraded.

Updating all 24 encoder layers gave the lowest error rates but required 230.4 million trainable parameters. Freezing layers lowers compute needs when data or memory is limited.

Two inference-side changes helped further without retraining. A larger attention context of 13 lookahead frames plus beam-8 MALSD decoding lowered WER by 2.71 absolute points. NVIDIA says this adds approximately 800 ms of latency, which suits batch transcription rather than live use.

The post also describes NVIDIA Nemotron 3 Diarization, which extends the pipeline to speaker-attributed transcription for up to eight speakers by aligning speaker boundaries with ASR timestamps.

What it means for developers

NVIDIA positions the workflow for teams that have enough labeled speech to specialize a model but not enough to train one from scratch, such as dialect adaptation, domain-specific transcription, or deployments that must keep existing languages. The baseline experiment used two NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs for 12,000 steps.

The company is careful about limits. These techniques are not universal defaults, replay protects only what its data represents, and partial unfreezing needs re-tuning when the data mix changes. The workflow also does not serve as evidence for every Arabic dialect or deployment environment. NVIDIA points readers to an ASR fine-tuning notebook, a Nemotron ASR fine-tuning skill, and a Hugging Face blog on adding diarization.