Together AI introduces compiler that continually optimizes AI inference

A compiler built for changing workloads

Together AI says inference workloads now span thousands of combinations of models, hardware platforms, customers, use cases and traffic patterns. The company serves many trillions of tokens per day across thousands of customers, whose workloads can differ in batch size, sequence length, mixture-of-experts imbalance, attention sparsity, session duration and domain.

The Together Compiler, also called TKCC, takes a workload's traffic shape, model and hardware as inputs. It then uses multiple agents to generate and test optimizations for an inference engine. Together AI describes TKCC as the first continual RSI system for this task, meaning it can repeatedly research, search and improve its own optimization choices as conditions change.

The system is intended to support a wide range of hardware, including H100s, B200s, B300s, NVL72, VR and Groq platforms. Together AI also says TKCC generalizes beyond language model inference to weather modeling, partial differential equations, time-series workloads and biology.

Persistent engines speed up experimentation

A central part of TKCC is its use of persistent engines. Instead of repeatedly starting and shutting down systems such as vLLM, SGLang or TRT-LLM for every experiment, the engine starts once and hot-reloads candidate forward passes without restarting.

According to Together AI, the conventional process in its comparison takes 467 seconds to reach serving, including weight loading, KV-cache sizing, JIT kernel warm-up and autotuning. Together Compiler takes 9.0 seconds in the same table. The company says this design produces 252x higher experiment throughput than baseline agentic RSI performance optimization systems and reduces GPU idle time while agents work.

TKCC also uses a tier of engines to balance fast experimentation with realistic measurement. Fast engines load a few layers for rapid, low-variance timing, while slower engines load many layers to test optimizations against real data distributions. This matters for architectures with data-dependent behavior, such as mixture-of-experts and sparse attention.

Reported results across models and domains

Together AI says TKCC generated 1.7x higher TPS than available open-source megakernels on the Llama 1B batch size 1 workload. On the SemiAnalysis InferenceX AgentX benchmark, the company reports 1.3-3x higher TPS than highly optimized baseline configurations across frontier models, inference engines and hardware platforms.

For low-latency batch size 1 tests, Together AI reports 1111 TPS on MiniMax M3.1-Flash and 1205 TPS on DeepSeek v4.1-Flash. The company says these results set the state-of-the-art on SemiAnalysis' InferenceX benchmark. It also lists Kimi K3, GLM 5.3 and DeepSeek v4.1-Flash among the models used in its broader evaluation.

Together AI says TKCC is already serving trillions of tokens. Its approach is designed to keep adapting as traffic shifts, including longer prompts after model releases, regional differences in prompt lengths and sudden request bursts that can affect cache hit rates. For inference providers and users, the potential significance is a system that can tune deployments continuously instead of relying on one manually optimized configuration.