Arm SME2 Cuts Streaming Speech Recognition Time on Pixel 11 by Up to 2.12x
A PyTorch case study finds that Arm's SME2 matrix extension cuts whole-recording processing time for the streaming speech model Nemotron 3.5 on a Google Pixel 11 by 2.12x with FP16 and 1.61x with INT8 (QC8W). The authors use the results to argue that chunk size and precision should be chosen together.
- Nemotron 3.5 was exported to ExecuTorch and profiled on a Google Pixel 11 with one inference thread.
- SME2 cut processing of a 24.9-second recording from 9.79 to 4.61 seconds for FP16 (2.12x) and from 5.20 to 3.24 seconds for QC8W (1.61x).
- From 80 to 1,120 ms chunks, FP16 WER fell from 15.2% to 11.5%, and QC8W from 15.5% to 11.7%.
- With SME2, real-time factors are 0.19 for FP16 and 0.13 for QC8W.
- Data movement now takes 17% of run time in both precisions and is a target for further optimization.
Why streaming ASR needs its own evaluation
The PyTorch post starts from a simple point: users expect text to appear while they speak. That makes streaming automatic speech recognition different from batch transcription, where the model sees the full recording first. Strong batch accuracy does not necessarily carry over to streaming.
The authors cite a cross-model ASR study as an example. Qwen3-ASR-1.7B had the best batch accuracy among the tested models, but its word error rate (WER) rose by 77% with 2.4-second audio chunks. In the same study's streaming comparison, NVIDIA Nemotron Speech Streaming 0.6B had the lowest average streaming WER over 560 ms chunks. The lesson, according to the post, is that accuracy must be measured at the chunk cadence used in deployment.
The case study takes the newer multilingual Nemotron 3.5 from PyTorch to ExecuTorch, PyTorch's edge inference framework, and profiles it on a Pixel 11.
How the model runs and what chunk size changes
Nemotron 3.5 is built around an RNN-T architecture with an encoder (FastConformer), an LSTM predictor and a small feedforward joiner. The encoder reduces time positions by a factor of 8, so each output position covers 80 ms of audio. At lookahead 13, a call collects 1,120 ms of audio and produces 14 encoder positions. Lookaheads 0, 3 and 6 give 80, 320 and 560 ms chunks. Longer chunks mean a longer wait for the first text and for later updates.
On the software side, ExecuTorch exports a model to a .pte file. Supported operations go to XNNPACK, and on Android XNNPACK can select microkernels from Arm KleidiAI. Operators not delegated to XNNPACK use ExecuTorch's portable CPU kernels.
Accuracy was measured on a host machine with the XNNPACK backend, using the Hugging Face Open ASR Leaderboard harness. Each data point covers 14,019 utterances from four corpora. Longer chunks improved final-transcript accuracy: from 80 to 1,120 ms, FP16 WER fell from 15.2% to 11.5%, and QC8W from 15.5% to 11.7%. FP16 differed from FP32 by less than 0.03 percentage points at every chunk size. QC8W, which uses dynamic INT8 activations and per-channel INT8 weights, added 0.21 to 0.31 percentage points of WER relative to FP32.
SME2 results on a Pixel 11
For the speed test, the team processed a 24.9-second recording with 1,120 ms chunks and one inference thread pinned to a mid-core CPU. They compared two ExecuTorch 1.3.1 builds, with SME2 on and off, both keeping XNNPACK and KleidiAI.
- FP16: 9.79 seconds down to 4.61 seconds (2.12x faster).
- QC8W: 5.20 seconds down to 3.24 seconds (1.61x faster).
With SME2 enabled, the real-time factors are 0.19 for FP16 and 0.13 for QC8W. The authors say this leaves headroom for other CPU work between chunks and can help manage thermal load, though it does not remove the delay of collecting audio.
ETDump operator timings show where the gain comes from. GEMM time fell from 8.2 to 3.1 seconds for FP16 and from 4.1 to 2.0 seconds for QC8W. GEMM is still the largest part of the run, but its share dropped from 84% to 67% for FP16 and from 78% to 62% for QC8W. Data movement takes about the same time in both builds, so its share rose from 8% to 17% for FP16 and from 9% to 17% for QC8W. The post calls this remaining work a target for further optimization.
Arm Streamline profiling on the Pixel 11 measured 38.4% SME utilization for FP16 and 53.7% for QC8W. The authors stress this shows how much of each capture the matrix hardware was allocated, not a percentage of peak arithmetic throughput.
What it means for developers
The study suggests developers should judge on-device speech recognition on accuracy and latency together rather than on batch scores. Longer chunks lower WER but delay text, while FP16 and QC8W trade a small accuracy difference for different processing times. On the tested phone, SME2 gave both precisions large processing headroom on a single thread. The post also notes that enabling SME2 only makes the paths available, so native profiling is needed to confirm which kernels actually ran.
