Dictation Cleanup Benchmark: Fast Small Models Trade Score for Speed
AssemblyAI's Voice Standard Bench puts Gemini 4 Argon first for dictation cleanup at both 30 seconds and 2 minutes, but the company argues that latency and exactness, not raw capability, should drive model choice. Smaller models answer much faster and cost a fraction of a cent per hour, at the price of lower scores.
- Gemini 4 Argon tops AssemblyAI's dictation cleanup bench with scores of 100 at 30 seconds and 99 at 2 minutes.
- Qwen3.5 4B Fast is the fastest pick, at a 0.28 s median at 30 seconds, but scores 81 and then 75.
- Nemotron 3 Nano 30B A3B is cheapest at $0.0013/hr at 30 seconds, but scores 67 at 2 minutes.
- Dictation is graded mechanically on a synthetic dataset, so AssemblyAI says to test shortlisted models on real recordings.
What the benchmark tests
According to AssemblyAI, dictation cleanup is "the one LLM task where the most capable model is often the wrong pick." A user stops talking and expects finished text almost at once, so a model that scores slightly higher but takes three times as long loses, the company says.
Voice Standard Bench is an open-source benchmark that runs every model on AssemblyAI's LLM Gateway (50+ at the time of writing) across four voice tasks: summarization, information extraction, text correction and dictation cleanup. The dictation task isolates one job: executing spoken formatting commands. Every model receives the same raw transcript in which punctuation, capitalization and formatting are spoken aloud, such as "comma," "new paragraph" or "all caps," and must return the formatted text.
The rules are strict. Spoken numbers stay as words, no word is added, dropped or reordered, and every command word is consumed. Leaving a literal "comma" in the text counts as a failure, as does rewriting the message into something more polished than what was said. The task does not score filler removal or self-corrections.
How scoring works
The three other tasks are graded by two LLM judges, Claude Opus 5.5 and GPT-5.6 Sol. Dictation has no judge. Because each input has a known correct answer, the bench grades it mechanically using:
- Exact match against the expected text
- Word error rate
- Punctuation accuracy
- Capitalization accuracy
- Formatting command accuracy
The dictation dataset is synthetic, with LLM-generated samples and a known truth text, 20 per length bucket. It runs only at the 30-second and 2-minute buckets, since nobody dictates for an hour into a text field. Each model also gets a median response time and a cost per hour of audio at provider list price.
Results at 30 seconds and 2 minutes
The live results page shows three cards per length. Figures below are as of October 28, 2026, according to AssemblyAI.
- 30 seconds: Highest Performance is Gemini 4 Argon with a score of 100, $0.047/hr and 2.00 s. Fastest is Qwen3.5 4B Fast with a score of 81 and a 0.28 s median. Lowest cost is Nemotron 3 Nano 30B A3B with a score of 81 and $0.0013/hr.
- 2 minutes: Gemini 4 Argon scores 99 at $0.068/hr and 2.61 s. Qwen3.5 4B Fast scores 75 with a 0.60 s median. Nemotron 3 Nano 30B A3B scores 67 at $0.0017/hr.
The gap between the top model and the small ones widens with longer input. AssemblyAI notes that the Fastest and Lowest cost cards have no quality floor, so the score should be checked before shipping either one.
The full table holds other options. At 30 seconds, Gemini 2.5 Flash Lite scores 97 for about $0.002 per hour of audio with a 0.70-second median. At 2 minutes it drops to 82.
Latency first, then exactness, then cost
AssemblyAI advises reading response time before quality. A dictation flow is interactive, and the bench's own guidance is that scores within 2 to 3 points differ only marginally. The company also warns that large, creative models sometimes over-edit, for example turning "twelve thirty four" into "12:34" when the rule says to keep words, and mechanical scoring penalizes that.
Cost is described as real but small: at 30 seconds to 2 minutes of audio, input and output run to a few hundred tokens, so absolute numbers stay low even though the spread between models is wide.
What it means for developers
AssemblyAI recommends filtering the Gateway model list by task and length, shortlisting two or three models and testing on real data. It cautions that the dictation data is synthetic and scored on exact match, so the bench should be treated as a shortlist, not a final verdict. For teams that do not want to pick a model, the company points to its Dictation API, which runs transcription and cleanup in one call and returns both the verbatim transcript and the rewritten text.
