AssemblyAI: Standard WER Benchmarks Can Penalize Better Transcription Models

Customers saw worse scores for a newer model

In a blog post by Zackary Klebanoff, Applied AI Lead, AssemblyAI describes an unexpected result after the launch of Universal-3 Pro, which it calls its most advanced transcription model to date. Some customers told the company that their benchmarks showed the new model performing worse than older AssemblyAI models. According to the company, that did not match its internal results, so it investigated.

The typical WER workflow, as the company outlines it, starts with 10 to 20 audio files representative of a customer's core use cases. A human transcription provider produces a reference transcript, known as the truth file. The same audio goes to several speech-to-text vendors. Both sets of transcripts are then normalized with the open-source Whisper Normalizer, which lowercases text, strips punctuation and maps equivalent words to a common form. Finally, a package such as jiwer compares them. WER is the percentage of words transcribed incorrectly, and lower is better.

Insertions turned out to be correct words

WER errors fall into insertions, deletions and substitutions. When AssemblyAI reviewed customer benchmarks in detail, it saw a disproportionate number of insertions, meaning words the model produced that were absent from the human truth file. The company listened to the original audio and says the vast majority of those insertions were correct. The words had been spoken, but the human transcriptionist had missed them.

According to AssemblyAI, on difficult audio such as noisy environments, accented speech, overlapping speakers and fast talkers, its model frequently outperformed the human-labeled transcript it was being scored against. The company's conclusion is that if the ground truth is wrong, the evaluation is wrong, and the problem becomes more visible as models improve.

Formatting differences also count as errors

A second issue involves semantic equivalence. AssemblyAI says Universal-3 Pro has a large language model decoder at its core, which gives it a stronger command of grammar and formatting. As a result, it may write "alright" where a human wrote "all right", "healthcare" instead of "health care", or "setup" instead of "set up". The meaning is identical, but a WER evaluation registers each as an error. The Whisper Normalizer covers some cases, such as don't and do not, but not every domain-specific or stylistic variation.

The correction tool and word lists

Because auditing every error by hand is unrealistic for hour-long recordings, AssemblyAI built a tool with two functions. The first is truth file correction. Users submit their truth files and audio, and the tool runs a Universal-3 Pro transcription and lists each discrepancy, with the matching audio segment playable inline. For each one, the user chooses among three options:

• AssemblyAI got it right, so the truth file is updated to match.
• AssemblyAI got it wrong, so the truth file stays as is.
• Neither is correct, so the user writes in what was actually spoken. Unclear segments can be labeled [inaudible].

The WER updates live as the user works through the errors. According to the company, this only needs to be done once, and the corrected truth files can be reused for future benchmarks, including comparisons against other vendors.

The second function is semantic word list generation. Users build groups of words to be treated as equivalent and port the list into their evaluation pipeline. AssemblyAI describes it as a simple find and replace for words that are semantically the same, intended so that stylistic differences do not unfairly penalize any vendor. The tool is available in the company's playground.

Streaming, the GitHub repo and what it means

AssemblyAI also advises evaluating async and streaming models separately. Async benchmarking means sending a full audio file and receiving a complete transcript. Streaming requires feeding audio bytes in real time, so the benchmark must run in parallel for the full duration of each file, which the company calls significantly more complex. Universal-3 Pro is available in both forms, and results should not be mixed.

The company is also releasing a GitHub repository showing how to build an internal benchmarking pipeline using corrected truth files and semantic word lists. It can support scheduled evaluations, WER tracking in CI/CD pipelines, or competitive benchmarks across several vendors.

For teams comparing transcription vendors, the message is to check the reference transcripts before trusting a ranking. AssemblyAI still calls WER a useful metric, but argues that uncorrected truth files and formatting mismatches can distort results, particularly for newer models. Note that the findings and the tool come from a vendor with an interest in the outcome, and the claims about Universal-3 Pro's accuracy are the company's own.