AssemblyAI Ranks LLMs for Pulling Structured Data From Spoken Transcripts
AssemblyAI has published a guide to choosing an LLM for extracting structured data from spoken transcripts, based on its Voice Standard Bench. The results show that the best model changes sharply with transcript length, and that small, fast models lose ground on longer recordings.
- AssemblyAI's Voice Standard Bench scores 50+ LLM Gateway models on extracting structured JSON from spoken transcripts at four lengths.
- Scoring combines a mechanical schema check with two LLM judges grading groundedness and completeness.
- As of October 28, 2026, the top scorer is gpt-5.6-sol at 30 seconds (94) and gpt-6.1-sol at 60 minutes (85).
- The 2-minute leader, nemotron-lightning-3p5-30b-a3b (100), is flagged as an outlier due to 15 errored requests and an 18.1 s median response time.
- Small fast models lose accuracy on long recordings: only 9 models score 80 or higher at 60 minutes.
Why transcripts are harder than documents
AssemblyAI argues that most advice on structured data extraction concerns PDFs, invoices and web pages, while voice products deal with transcripts of people talking. The company names four problems specific to speech: misheard and spelled-out numbers, spelled names and hedged dates, speaker attribution, and cross-talk and disfluency.
On a two-party call, for example, the callback number may be a wrong value the agent read back, not the right one the caller stated. Without speaker labels, extraction can pull the wrong turn. According to AssemblyAI, a model trained to produce fluent text can also smooth over a gap by inventing something plausible, which it calls the worst outcome in extraction. For that reason the bench's rubric leads with groundedness.
How the benchmark scores extraction
Voice Standard Bench runs every model on AssemblyAI's LLM Gateway (50+ at the time of writing) on real speech-to-text output across four tasks: summarization, information extraction, text correction and dictation cleanup. For extraction, the model must fill one fixed JSON shape with participants, action items (each with an owner, a task and a due date), decisions, and named entities. Every key must be present. Empty fields get an empty array, and an unstated owner or due date must be marked "unspecified", not null and not a guess.
Scoring has two steps. A mechanical schema check comes first, and outputs that fail are never judged and receive the worst possible rubric score. Valid outputs are then graded by two LLM judges, Claude Opus 5.5 and GPT-5.6 Sol, on groundedness and completeness. The task runs at four lengths: 30 seconds, 2 minutes, 15 minutes and 60 minutes, with 10 real transcripts per bucket. Each model also gets a median response time and a cost per hour of audio at provider list price. Requests that error out are recorded but left out of the score.
Top picks by transcript length
The live results page shows three cards per length: Highest Performance, Lowest cost and Fastest. As of October 28, 2026, AssemblyAI lists these picks:
- 30 seconds: gpt-5.6-sol leads with a score of 94. nemotron-3-nano-30b-a3b is cheapest (score 73, $0.0015 per hour of audio), and qwen3.5-4b-32k-fast is fastest (score 80, 0.28 s median).
- 2 minutes: nemotron-lightning-3p5-30b-a3b tops the view with 100, but AssemblyAI calls it an outlier. The bench logged 15 errored requests, its median response time is 18.1 seconds, and it scores 26 at 30 seconds. The next-highest scores are gpt-5.6-luna and gemini-4-argon, at 91 each.
- 15 minutes: gpt-6-luna leads with 85. nemotron-3-nano-30b-a3b is cheapest (score 49, $0.0013 per hour), and qwen3.5-4b-32k-fast is fastest (score 33, 1.49 s median).
- 60 minutes: gpt-6.1-sol leads with 85. nemotron-3-nano-30b-a3b is cheapest (score 52, $0.0012 per hour), and gemini-3.5-flash-lite is fastest (score 59, 2.86 s median).
The Lowest cost and Fastest cards have no quality floor, so AssemblyAI advises checking the score before shipping either. Scores within 2 to 3 points of each other differ only marginally.
What changes as recordings get longer
On voicemails and short calls, facts are few and close together, so small models find them quickly. At 30 seconds, 43 of the 52 models on the tab score 80 or higher. On a 15-minute support call, an order number may appear once, be corrected ten minutes later, and end up in the agent's read-back, which is where completeness starts to separate models. At 60 minutes, context handling dominates: some models drop facts from the middle of a long transcript, and only 9 models score 80 or higher.
The fastest model illustrates the shift. qwen3.5-4b-32k-fast scores 80 at 30 seconds and 33 at 15 minutes. AssemblyAI says its launch post's finding that "most voice tasks don't need frontier models, or frontier-model prices" still holds, but that the exception is most visible on this task at the longest buckets.
What it means for teams building voice products
AssemblyAI suggests four questions before picking a model: how strict the schema is, how costly a hallucinated field would be, what the latency budget is, and how long the transcripts are. If downstream code parses the output, it recommends models that support structured outputs through response_format, and validating every response anyway. It advises weighting groundedness over completeness, and giving the model a legitimate way to say "not stated".
For routing a voice agent mid-call, where a response is needed in well under a second, the Fastest card is the starting point. Nightly extraction over past recordings can use the top performer regardless of speed. The company says the live page is the source of truth, since rankings change as models change.
