AssemblyAI bench: small LLMs match big models on short audio, fade at meeting length

What the bench found about cost and length

AssemblyAI says a small, fast LLM answers a 30-second voicemail or a 2-minute call in about a second and lands within a few points of the largest models. On an hour-long meeting, the larger model is worth paying for. The company says this comes from measurement, not intuition.

In the current Voice Standard Bench run, the top-scoring model costs 15 to 77 times more per hour of audio than a model within two quality points of it. On hour-long transcripts that premium narrows to about 4×, which AssemblyAI describes as the point where bigger models earn it.

The bench runs every model on LLM Gateway against real speech-to-text output at four lengths: about 30 seconds, 2 minutes, 15 minutes and 60 minutes. Scores run from 0 to 100, with every task weighted equally. In the run published October 13, 2026, the top pick at each length scored 96, 94, 89 and 88. GPT-6 Luna scored 95, 92, 87 and 85. gemma-4-31b scored 93, 89, 64 and 63. Qwen3.5 4B Fast scored 74, 64, 47 and 31. AssemblyAI notes that scores within 2 to 3 points are effectively tied.

Qwen3.5 4B Fast: the fastest, with hard limits

Qwen3.5 4B Fast is the open-source Qwen3.5 4B model, hosted by AssemblyAI on its own GPUs in a latency-optimized setup with a 32,768-token context window. It costs $0.10 per million input tokens and $0.50 per million output tokens.

It is the fastest model in the bench at every length up to 15 minutes, with median times of 0.28 seconds at 30 seconds, 0.60 seconds at 2 minutes and 1.49 seconds at 15 minutes. In launch testing on voice rewrite tasks, AssemblyAI says it averaged 612 ms, 1.9× faster than GPT-4.1 at 94% lower cost per hour of audio. On 2-minute transcripts it costs about $0.003 per hour of audio.

The company positions it for short rewrite work where a user is waiting: dictation cleanup, turning transcripts into emails or notes, live formatting and compressing a conversational turn into one line. The quality trade is real. At 30 seconds it scores 62 on text correction and 81 on dictation, against 93 and 98 for gemma-4-31b. AssemblyAI suggests it when the latency budget is under about 500 ms.

Its information-extraction score falls from 80 at 30 seconds to 51 at 2 minutes and 33 at 15 minutes. Through the API it supports only max_tokens, temperature and stream, with no tool calling or structured output, so AssemblyAI calls it the wrong pick for JSON extraction or voice agents. The company hosts the model and says it publishes these weaknesses anyway so teams can route traffic well.

Other small open models and their trade-offs

AssemblyAI also looks at three open models that hold up longer. The ones in the post are served through AWS Bedrock, DigitalOcean or Fireworks.

• gemma-4-31b answers a 30-second input in under a second at a score of 93, against 1.7 seconds for GPT-6 Luna, and a 2-minute call in 1.5 seconds against 2.4. It is the lowest-price model within 8 points of the top for text correction (84), but falls to 64 at 15 minutes.
• GLM-5.3 Flash is the lowest-price model within 8 points of the top for summarization (91) and dictation (96). At 15 minutes it scores 74, well behind the leaders.
• Nemotron 3 Nano 30B A3B costs about $0.002 or less per hour of audio at every length. Its information-extraction scores are 73, 58, 49 and 52 across the four lengths. AssemblyAI's takeaway is that the cheapest model is not the cheapest choice if it cannot do the job.

The company also explains that mixture-of-experts models like Nemotron have 30 billion parameters in total but activate only about 3 billion per token, so they compete with small models on price and speed.

Why length hurts, and why hosting matters

AssemblyAI rules out the obvious explanation. An hour-long meeting is well over 10,000 input tokens, which still fits inside Qwen3.5 4B Fast's 32,768-token window. The scores instead point to how much the task asks a model to keep track of. Summarization coverage and extraction completeness reward capturing topics, decisions, action items and named entities.

A voicemail has one caller and one request. A long meeting has dozens of decisions and owners, with topics that return 40 minutes later. Every dropped item costs points, and the company says smaller models drop more as transcripts grow. It adds two caveats: this is what the scores show, not a full account of the mechanism, and dictation was measured only at 30 seconds and 2 minutes. Teams should test on their own long transcripts.

Hosting also matters. In the bench, the same open-model families hosted elsewhere take 10 to 20 seconds to respond, so AssemblyAI says Qwen3.5 4B Fast's sub-second speed comes from a tuned deployment and not only from its 4 billion parameters. The hosting provider is also the one that processes the transcript, so data location is part of the choice.

What it means for teams routing voice traffic

The practical message from AssemblyAI is to route by transcript length. For short, latency-sensitive rewrites, small models are close to the leaders at a fraction of the cost. For meeting-length summarization and extraction, the gap grows and larger models justify their price. The source also notes that the best value in the bench is not an open model: GPT-6 Luna scores 95, 92, 87 and 85 across the four lengths. Prices cited are list prices as of October 20, 2026, and AssemblyAI points readers to its pricing page and docs for current rates and provider details.