Baseten Serves Qwen3-ASR 7x Faster for Batch Transcription

Batch transcription is a throughput problem

Baseten separates speech-to-text into two workloads. Real-time transcription is about latency, since a voice agent needs words back within a few hundred milliseconds. Batch transcription is about throughput: podcast networks, meeting-notes products and call-center archives hold hours of recorded audio and want it transcribed as cheaply as possible, which means pushing as many audio hours as possible through each GPU.

The company says it optimized Qwen3-ASR 1.7B on its platform to reach the lowest cost per audio minute across all transcription providers. Qwen3-ASR 1.7B is the Qwen team's open-weight speech recognition model, released under an Apache 2.0 license. It supports more than 30 languages and 22 Chinese dialects, and sits on the Pareto frontier of Hugging Face's Open ASR Leaderboard with a 5.4% average word error rate (WER). On that leaderboard it scores 5.40%, within 0.07 points of NVIDIA Nemotron 3.5 Transcribe 30B-A4B (5.33%), the most accurate model, at a fraction of its size.

Four areas of optimization

Baseten started from vLLM's OpenAI-compatible transcriptions endpoint, which chunks audio into 30-second windows and batches them through the model. It then profiled each stage of a request under load and focused on four areas.

• Native audio decoding: vLLM's default loader uses PyAV, which decodes frame by frame in Python and holds the global interpreter lock, serializing decode threads. Baseten replaced it with torchcodec, which passes the whole file to FFmpeg in one native call. Long files are also decoded in parallel across time ranges with small overlaps. The company says an hour of audio can be preprocessed in well under a second. In its measurements, the Python decoder took 76 ms for a one-minute file alone but 2,195 ms when 8 files decoded at once, while the native decoder took 45 ms alone and 48 ms with 8.
• KV cache quantization: Attention over the KV cache is the largest share of GPU time at high concurrency. Storing the cache in FP8 halves the bytes read per step and doubles the number of sequences that fit in memory. Because vLLM's default FlashAttention-2 backend does not support FP8 cache on RTX PRO 6000, Baseten used FlashInfer kernels, which raised memory utilization for the attention kernel from 50% to 80%. Weights stay in BF16, since weight quantization increased WER on the company's test sets.
• Host-to-device syncs: The attention backend needed each sequence's length from the GPU, and the audio encoder read back the maximum clip length once per layer, both forcing the CPU to wait. Baseten keeps host-side copies of these values instead. GPU utilization under load rose from 86% to 97%, and the engine went from CPU-bound to GPU-bound.
• Request handling: Four API-server processes run ahead of the engine so decoding and log-mel feature extraction proceed in parallel, and the preprocessing thread pool in each process was doubled.

Throughput and latency results

All comparisons use a single RTX PRO 6000 with the same weights, testing upstream vLLM with no changes against Baseten's implementation at 64 concurrent requests. Vanilla vLLM plateaus around 380x real-time factor (RTF), while the optimized stack reaches 2,645x on 5-minute clips and 3,128x on hour-long files. The gain grows with clip length, from 4x on 10-second clips to 7x on 5-minute clips. Baseten could not test vanilla vLLM on audio beyond 10 minutes because such files do not fit in one request.

Latency also improved. For 5-minute clips, a single request fell from 5.1 seconds to 1.5 seconds, and at 64 concurrent requests the median dropped from 50 seconds to 7.3 seconds, which Baseten describes as 6.9x lower latency. Files that callers previously had to split now return in one request: 3.0 seconds for 30 minutes of audio and 4.7 seconds for a full hour.

What it means for users

For teams processing large audio archives, the figures point to lower per-hour costs and fewer workarounds, since long recordings no longer need to be split before submission. Baseten says Qwen3-ASR has been served for batch transcription on its platform since the model launched, and that its real-time deployment shows the lowest WER and lowest latency on Coval's speech-to-text benchmark. The performance numbers are the company's own measurements against an unmodified vLLM baseline.