Together AI outlines self-tuning approach to open-model inference

Inference designed to change shape

Together AI describes Polymorphic Inference as inference in “many shapes”, a response to the faster release of open-source models and the growing variety of ways they are deployed. The company says that 80% of traffic on OpenRouter goes to models released in the last three months, illustrating the pace that inference systems must handle.

The approach also reflects architectural diversity. According to Together AI, all of the top-ten most popular open-source models in use today use different architectures. Traffic can change as products, user behavior, geographies and service-level agreements evolve, while model post-training can alter how many tokens a system uses for tasks such as reasoning.

Hardware is changing as well. Together AI points to shifts in inference-serving capabilities from B200 to GB300 and says the industry is approaching the first Vera Rubin deployments, alongside changes in CPUs, storage systems and alternative inference chips.

Why static optimization falls short

Together AI contrasts Polymorphic Inference with what it calls “Static Inference”, an earlier model of serving based on closed models, traditional Transformer architectures, single-user workloads and a single hardware platform. In that environment, an inference engine could be tuned around one configuration and expected to perform consistently.

The company argues that this assumption is increasingly unreliable. A configuration optimized for one workload may not be the best choice for another, and open-source kernels built around fixed traffic shapes can be poorly matched to production traffic. Together AI gives mixture-of-experts kernels as an example, saying that performance on random tokens may not transfer to real production settings.

The company also says its experience deploying megakernels changed its view of the best optimization target. A megakernel can be designed around one static shape, such as batch size one for Llama 1B, but Together AI now argues that model architectures and workloads require different technical approaches.

Together Compiler brings continuous optimization

Together Compiler is presented as an early implementation of the company’s Polymorphic Inference agenda. Together AI describes it as a rapid, AI-powered performance optimization loop that changes inference optimization from a traditional compiler translation task into an agentic search problem.

The system identifies key optimization patterns and pairs a swarm of agents with a persistent engine for rapid iterations. Together AI says Together Compiler already powers Continuous Compilation optimization loops across many of its top production inference workloads and is running on some of its most popular serverless endpoints, where it has found additional gains on top of already optimized code.

According to the company, this approach can reduce the need to create new hand-written kernels and optimizations whenever traffic shapes or workloads change. Together Compiler is available to end customers in limited beta today, and the company says customers will soon be able to use it to optimize dedicated endpoints for their own models, engines, traffic shapes and use cases.

Together AI also previews work on shape-shifting speculative decoding, dynamic routing across heterogeneous deployments through ThunderAgent Router, and further research informed by serving trillions of tokens per day. For users of open models, the stated goal is inference that can adjust more quickly as models, applications and computing platforms change, rather than relying on a single fixed configuration.