Best LLM for Summarizing Transcripts Depends on Length, AssemblyAI Bench Finds

Four lengths, four different answers

According to AssemblyAI, most summarization advice focuses on documents, while transcripts differ because the input is spoken: disfluent, repetitive, with two or more speakers and sometimes misrecognized words. Length also varies by two orders of magnitude within the same product.

The bench tests four buckets. At 30 seconds, almost any model gets the content right: 41 of the 52 models on the Summarization tab score 90 or higher, so speed, price and padding decide. At 2 minutes, faithfulness starts to matter. At 15 minutes, coverage separates models, and only 10 models score 90 or higher. At 60 minutes, the spread between the top and bottom of the ranking is widest, while the speed gap between small and large models narrows.

AssemblyAI says the broader pattern from the bench is that most voice tasks do not need frontier models or frontier-model prices. Open and small models are competitive on short inputs, and the gap opens at meeting length.

How the benchmark scores summaries

Voice Standard Bench is open source and runs every LLM Gateway model (50+ at the time of writing) on four tasks: summarization, information extraction, text correction and dictation cleanup. Every model gets the same prompt, asking for a summary a busy person could read in place of the call, stating only what was said, with length scaled to the call.

Outputs are scored by two LLM judges, Claude Opus 5.5 and GPT-5.6 Sol, on three criteria: faithfulness, coverage and conciseness. AssemblyAI argues that metrics such as ROUGE and BERTScore, which rely on token overlap with a reference, fit spoken input poorly. The company says the two judges disagree by a few points on average, that self-preference bias was measured and found small, and that scores within 2 to 3 points should be read as a tie.

Each bucket uses 10 real transcripts within 15% of the target length, produced by AssemblyAI Speech-to-Text. Errored requests are recorded but left out of the score rather than counted as zero, so a model with many errors can post a high score on the calls that completed.

The bench also reports cost per hour of audio at provider list price, rather than per token. On 2-minute calls, claude-opus-5-5 (score 99) costs about $0.30 per hour of audio, while nemotron-3-nano-30b-a3b (score 82) costs about $0.0016. AssemblyAI describes that as a 17-point quality gap at under 1% of the price.

The picks as of October 28, 2026

The live results page shows three cards per length: Highest Performance, Lowest cost and Fastest. AssemblyAI notes that Lowest cost and Fastest have no quality floor, so scores should be checked before shipping one.

• 30 seconds: gpt-5.6-terra scores 100. nemotron-3-nano-30b-a3b is cheapest (score 76, $0.0016 per hour). qwen3.5-4b-32k-fast is fastest (score 76, 0.28 s median).
• 2 minutes: claude-opus-5-5 scores 99. nemotron-3-nano-30b-a3b scores 82 at $0.0016 per hour. qwen3.5-4b-32k-fast scores 69 at 0.60 s.
• 15 minutes: claude-opus-5-5 scores 97. nemotron-3-nano-30b-a3b scores 77 at $0.0013 per hour. qwen3.5-4b-32k-fast scores 60 at 1.49 s.
• 60 minutes: deepseek-v4.1-flash scores 96. nemotron-3-nano-30b-a3b scores 69 at $0.0011 per hour. gemini-3.5-flash-lite is fastest (score 76, 2.86 s).

The 60-minute top score comes with a caveat. According to AssemblyAI, deepseek-v4.1-flash costs about $0.0092 per hour of audio, but the bench logged 25 errored requests across its 60-minute runs, so the 96 reflects completed calls only. Its median response time is 73.3 seconds. The next-highest scores are claude-opus-5-5 and gpt-6-astra, both at 95.

What it means for teams choosing a model

AssemblyAI's guidance is to match the model to the use. For a voicemail shown instantly in a UI, response time matters, but the fastest model trails the top by 24 points, so it suggests looking one or two rows down the speed ranking. For CRM notes after a 2-minute call, a low-cost model is usually the right default, provided its score is close enough to the top. For summaries that drive a follow-up such as a ticket or escalation, it recommends leaning toward the highest performer and accepting latency, which batch processing hides.

At 60 minutes, the company says model choice matters most, and that teams needing reliability should weigh error behavior on their own long calls as well as the score. For cost planning, it suggests multiplying the cost-per-hour figure by monthly audio hours: for a support team summarizing 10,000 hours a month, the gap between a top performer and a budget pick becomes a real line item. AssemblyAI adds that the live page is the source of truth and changes as models change.