Helion Brings Autotuned GEMM Kernels to vLLM, Beating CUTLASS and DeepGEMM on Hopper
The PyTorch team has integrated Helion, its Python-based kernel DSL, into vLLM's linear backend. According to the team, on NVIDIA Hopper GPUs it outperforms the vLLM default CUTLASS and DeepGEMM backends across the evaluated models, with more than 10% throughput improvement for some workloads.
- The PyTorch team added a Helion linear backend to vLLM, focused on NVIDIA Hopper GPUs.
- One Helion GEMM covers Standard, Split-K and Swap-AB, with variant and config picked by per-shape autotuning.
- According to the team, it beats the vLLM default CUTLASS and DeepGEMM backends across evaluated models, with over 10% throughput gains for some workloads.
- Targeted formats are FP8_Dynamic, W8A8_INT8 and Block_FP8.
- Challenges include hours-long tuning, cold-start latency, CPU launch overhead and config maintenance.
One kernel for several GEMM variants
The core idea is to replace several hand-written kernels with a single autotuned one. According to the PyTorch post, one Helion general matrix multiplication (GEMM) implementation can cover Standard GEMM, Split-K and Swap-AB. The best variant and its configuration are chosen automatically through per-shape autotuning.
The two specialized variants target particular input shapes. Split-K partitions the K dimension across multiple thread blocks to increase parallelism when the M and/or N dimensions are too small. Swap-AB rewrites A@B as ([email protected]).T to improve performance for shapes with a small M dimension, enabling more favorable tiling and better GPU utilization.
Traditionally, kernel authors implement each variant, benchmark them against the standard GEMM and write heuristics to decide which one to dispatch for a given shape. The post notes that vLLM's current Block_FP8 linear backend dispatches to the Swap-AB variant when M < 32 on Hopper, and that such heuristics may not be optimal for every model or workload. With Helion, the algorithmic choices become tunable parameters. In the example kernel, split_k is constrained to power-of-two values up to 256, and swap_ab is a boolean switch.
Scope: Hopper and quantized formats
The work focuses on NVIDIA Hopper GPUs and uses Helion's Triton backend. Because FP8 and INT8 are the primary quantization formats for efficient inference on Hopper, the team targets three quantized GEMM formats:
- FP8_Dynamic: FP8 per-token activation and per-channel weight scaling.
- W8A8_INT8: INT8 per-token activation and per-channel weight scaling.
- Block_FP8: FP8 with 1×128 activation scaling and 128×128 weight scaling.
The team says initial results with Helion's CuteDSL backend show competitive GEMM performance on NVIDIA Blackwell GPUs, and that this work can be extended to Blackwell as that backend matures.
The results rely on per-shape tuning combined with hybrid dispatch, according to the post, and deliver consistent end-to-end gains over the vLLM defaults.
Why Helion, and what it costs
vLLM is a high-performance inference and serving framework for large language models. For quantized linear layers it uses specialized backends that integrate kernels from libraries such as CUTLASS, DeepGEMM and FlashInfer. Helion is a PyTorch-native, hardware-agnostic kernel DSL built on a tile-programming model. Developers write a kernel once in Pythonic code, and an ahead-of-time (AOT) autotuner explores a broad search space, from memory layout and scheduling to algorithmic choices, to pick the best config. Helion also supports LLM-guided search to improve tuning efficiency.
The post argues that Helion treats tuning as a structured numerical optimization problem over a well-defined search space, which it says is more robust and reliable than open-ended agentic approaches that repeatedly generate, profile and refine kernels. It also says users could tune kernels for their own deployments without specialized kernel expertise or significant engineering effort.
The authors are open about the challenges:
- Tuning time: fine-grained tuning can still take hours.
- Startup time: CUDA Graph capture during vLLM startup triggers Helion JIT compilation, raising cold-start latency. Caching compiled artifacts can largely remove this on warm starts.
- Runtime CPU overhead: outside CUDA Graph capture, kernel dispatch and launch can add CPU overhead that may offset tuning gains. Helion is most effective under CUDA Graphs.
- Maintenance: shipping pre-tuned configs for popular models creates an upstream burden, since large config files are hard to maintain and validate through unit tests and CI.
The team frames these as a tradeoff among performance, usability and maintainability, where optimizing for two often comes at the expense of the third.
What it means for vLLM users
For teams serving quantized models on Hopper GPUs, the work points to a way to gain throughput without maintaining several separate GEMM kernels and dispatch rules. The price is tuning time and some operational care, such as running under CUDA Graphs and caching compiled artifacts. How much benefit a given deployment sees depends on the model and workload, since the reported gain above 10% applies to some workloads rather than all.
