Together AI says simpler kernels can beat its megakernels

Together AI says it has replaced its difficult-to-deploy AI megakernels with a more adaptable approach built around CUDA Graphs, PDL, Torch Inductor kernels and automated optimization. The company says this approach recovered 98% of megakernel performance without writing a single kernel, while multi-kernels and agent loops surpassed its previous systems by up to 1.7x.
- Together AI says CUDA Graphs and PDL recovered 98% of megakernel performance.
- Multi-kernels and agent loops reached over 1100 TPS for Llama 1B on H100.
- The company says its approach surpassed older megakernels by up to 1.7x.
- Inductor exposed 607 tuning flags as of Q3 2026, according to Together AI.
- Together AI found isolated benchmarks could misrepresent live inference performance.
From record speeds to production constraints
Together AI's announcement comes as AI inference workloads become more varied. Model architectures change frequently, traffic moves across different batch sizes and sequence lengths, and hardware platforms continue to proliferate. The company says optimization systems need to handle those changing conditions rather than target one fixed configuration.
Last year, Together AI helped release some of the first AI megakernels. These systems executed an entire model forward pass inside one highly fused GPU kernel and reached record-breaking speeds. Production deployment proved much harder, however. After months of work, the company says it had one model, one batch size and one customer.
That experience led Together AI to reconsider whether the largest possible fused kernel was always the right target. The company says that CUDA Graphs and PDL, combined with default Torch Inductor kernels, recovered 98% of megakernel performance without requiring a custom kernel. It also says that multi-kernels and agent loops beat its older megakernels by up to 1.7x, reaching over 1100 TPS for Llama 1B on H100.
Why Together AI is looking beyond compiler heuristics
Together Compiler is presented as a framework for continual AI inference optimization. In the company's account, the project grew from two years of work on GPU programming and optimization systems, including investigations into Torch Compile and the megakernels that followed.
Together AI says Torch Compile, powered by Inductor, captures PyTorch operations in a computation graph, optimizes that graph and generates executable GPU kernels. Inductor uses pairwise, greedy fusion decisions, with heuristics that estimate factors such as memory-traffic savings. The company argues that these local decisions do not search globally across all possible kernel partitions and execution strategies.
Its experiments on Llama 1B found that compiler settings can produce sharply different results depending on the workload. Together AI says enabling freezing improved throughput by 10, 26% at larger batch sizes, but caused a 49% slowdown at batch size 1. As of Q3 2026, the company says Inductor exposed 607 tuning flags, including 198 added during the previous year. Together AI argues that this scale makes systematic search more useful than manually guessing which flags to enable.
Benchmarking can miss real inference behavior
Together AI also questions whether isolated kernel benchmarks reliably predict live inference performance. The company says Inductor clears the GPU's L2 cache before measuring individual kernels, even though kernels in a real forward pass may reuse data left by preceding operations. In its experiments, isolated benchmark timings were 39, 58% slower than timings during live inference when cache reuse reduced memory traffic.
The company describes the opposite problem for embedding benchmarks. It says Inductor initialized token IDs to zero, causing every token to access the same embedding vector. For a 64K-token sequence, Together AI says this made the kernel 2.1x faster than when random token IDs were used, because the repeated accesses created unusually high cache reuse.
These findings support Together AI's broader case for an optimization framework that tests configurations in realistic execution contexts. The company says the remaining parts of its blog will examine the limitations of megakernels, the simpler primitives behind their gains and the agent-driven infrastructure used to shape Together Compiler.


