Artificial intelligence news.
Straight from the original announcements.

Together AI says simpler kernels can beat its megakernels

November 16, 2026By the AI News Bites Editorial TeamSource: Together AI
Together AI says simpler kernels can beat its megakernels
Image: Together AI

Together AI says it has replaced its difficult-to-deploy AI megakernels with a more adaptable approach built around CUDA Graphs, PDL, Torch Inductor kernels and automated optimization. The company says this approach recovered 98% of megakernel performance without writing a single kernel, while multi-kernels and agent loops surpassed its previous systems by up to 1.7x.

Key points
  • Together AI says CUDA Graphs and PDL recovered 98% of megakernel performance.
  • Multi-kernels and agent loops reached over 1100 TPS for Llama 1B on H100.
  • The company says its approach surpassed older megakernels by up to 1.7x.
  • Inductor exposed 607 tuning flags as of Q3 2026, according to Together AI.
  • Together AI found isolated benchmarks could misrepresent live inference performance.

From record speeds to production constraints

Together AI's announcement comes as AI inference workloads become more varied. Model architectures change frequently, traffic moves across different batch sizes and sequence lengths, and hardware platforms continue to proliferate. The company says optimization systems need to handle those changing conditions rather than target one fixed configuration.

Last year, Together AI helped release some of the first AI megakernels. These systems executed an entire model forward pass inside one highly fused GPU kernel and reached record-breaking speeds. Production deployment proved much harder, however. After months of work, the company says it had one model, one batch size and one customer.

That experience led Together AI to reconsider whether the largest possible fused kernel was always the right target. The company says that CUDA Graphs and PDL, combined with default Torch Inductor kernels, recovered 98% of megakernel performance without requiring a custom kernel. It also says that multi-kernels and agent loops beat its older megakernels by up to 1.7x, reaching over 1100 TPS for Llama 1B on H100.

Why Together AI is looking beyond compiler heuristics

Together Compiler is presented as a framework for continual AI inference optimization. In the company's account, the project grew from two years of work on GPU programming and optimization systems, including investigations into Torch Compile and the megakernels that followed.

Together AI says Torch Compile, powered by Inductor, captures PyTorch operations in a computation graph, optimizes that graph and generates executable GPU kernels. Inductor uses pairwise, greedy fusion decisions, with heuristics that estimate factors such as memory-traffic savings. The company argues that these local decisions do not search globally across all possible kernel partitions and execution strategies.

Its experiments on Llama 1B found that compiler settings can produce sharply different results depending on the workload. Together AI says enabling freezing improved throughput by 10, 26% at larger batch sizes, but caused a 49% slowdown at batch size 1. As of Q3 2026, the company says Inductor exposed 607 tuning flags, including 198 added during the previous year. Together AI argues that this scale makes systematic search more useful than manually guessing which flags to enable.

Benchmarking can miss real inference behavior

Together AI also questions whether isolated kernel benchmarks reliably predict live inference performance. The company says Inductor clears the GPU's L2 cache before measuring individual kernels, even though kernels in a real forward pass may reuse data left by preceding operations. In its experiments, isolated benchmark timings were 39, 58% slower than timings during live inference when cache reuse reduced memory traffic.

The company describes the opposite problem for embedding benchmarks. It says Inductor initialized token IDs to zero, causing every token to access the same embedding vector. For a 64K-token sequence, Together AI says this made the kernel 2.1x faster than when random token IDs were used, because the repeated accesses created unusually high cache reuse.

These findings support Together AI's broader case for an optimization framework that tests configurations in realistic execution contexts. The company says the remaining parts of its blog will examine the limitations of megakernels, the simpler primitives behind their gains and the agent-driven infrastructure used to shape Together Compiler.

Related news

Modular Rebuilds MAX Module for Faster Development and Inference
For developersNovember 19, 2026

Modular Rebuilds MAX Module for Faster Development and Inference

Modular has rebuilt the Module API in its MAX AI framework, combining faster eager execution during development with compiled production inference. The redesigned API…

Meta Introduces TLX, a Hardware-Native Compiler for GPU Workloads
For developersNovember 10, 2026, 17:59

Meta Introduces TLX, a Hardware-Native Compiler for GPU Workloads

Meta has detailed Triton Low-Level Extensions, or TLX, an extension to Triton designed to improve GPU performance in production machine learning workloads. TLX gives…

Together AI introduces compiler that continually optimizes AI inference
For developersNovember 4, 2026

Together AI introduces compiler that continually optimizes AI inference

Together AI has introduced Together Compiler, a continuous compilation system designed to optimize AI inference as models, hardware and customer traffic change. The…