Deepgram's Nova-3 on AWS Trainium2 processes 11, 70% more audio per dollar

Why Deepgram targeted inference efficiency

According to AWS, Deepgram's models cover speech-to-text, text-to-speech and voice agents, and they now process quadrillions of audio tokens a year for more than 200,000 developers and 1,400 organizations. Customers use them through the Deepgram API or run them in their own environments, including on Amazon SageMaker AI.

For Deepgram, serving efficiency decides how much audio the system can handle for a given infrastructure budget. High-volume batch workloads put heavy pressure on that budget, so the more efficiently each accelerator hour is used, the more audio can be served on the same fleet. The companies focused on offline batch uses such as contact-center analytics and medical transcription.

The final configuration runs eight model replicas on a single Amazon EC2 trn2.3xlarge instance. AWS says the price-performance figure was measured using instance pricing at the time of the evaluation.

From PyTorch to tuned kernels

The project started with Nova-3 delivered as standard PyTorch code. Over several months, the teams ported it to Trainium2 using TorchNeuron, the native PyTorch integration of the Neuron Compiler. The model first ran in eager mode to find operations that fell back to the CPU. After compiling with torch.compile, every model operation ran on Trainium2 with no CPU fallbacks detected. The encoder and decoder were compiled as separate static-shape graphs. Accuracy was checked at layer level against CPU and through end-to-end Word Error Rate, which had to stay in line with Deepgram's reference values.

The team then swapped in fused kernels from the public NKI Library, covering operations such as fused QKV projections, fused feed-forward layers and depthwise 1D convolution. Profiling with Neuron Explorer and the Neuron Agentic Development toolkit showed where hand-written kernels would do better:

• FFN: a hand-tuned kernel loads each weight tile once and reuses it, giving 2.0x lower latency than the reference kernel at batch size 96.
• Depthwise conv1d: rewritten as a multiply-accumulate loop on the Vector Engine, with a 1.5x speedup over the reference kernel at batch size 96.
• Conv2d: a new kernel written from scratch removed a compilation bottleneck beyond batch size 32, with up to 50x on the convolution module at batch sizes of 32 or less.

A Rust runtime for production serving

Deepgram's production serving stack is written in Rust, so the model had to run without Python or PyTorch in the serving path. The teams built a lightweight runtime in which Rust coordinates model replicas through a shared work-stealing queue, with one thread per replica. A C layer loads ahead-of-time-compiled NEFF files and runs them through the Neuron Runtime API.

Each Trainium2 device has eight physical NeuronCores, exposed by default as four logical cores. Because a Nova-3 instance fits on a single physical core, the team set one physical core per logical core, which allows eight replicas per device. According to AWS, the runtime reproduced the reference token sequences exactly at every tested batch size.

Compared with Python multiprocessing and multithreading baselines, the Rust runtime avoids interpreter overhead and global interpreter lock contention. This matters because Nova-3 makes many small decoder calls per audio chunk, so host-side dispatch limits throughput at low batch sizes. At the throughput-maximizing batch size, the Rust runtime delivered a 7.5% throughput gain over the multiprocessing Python baseline.

What it means for customers

AWS says the savings help Deepgram scale efficiently, keep shipping better models and offer them at lower prices. The benefits may also reach customers who self-host Deepgram models in their own VPC on Amazon SageMaker AI, where the two companies continue to work on setup, reliability and observability. The reported gains are specific to offline batch workloads, and throughput results are given as relative gains rather than absolute audio processed per second.