Open-Weight Models Move Into Production, Kilo Panel Hears

From cost to control

The panel brought together three layers of the stack: the agent harness, the model and the inference engine. Ari Messer of Anaconda (Kilo) said interest in open models has existed for some time, especially in regulated sectors that care about visibility and governance. What is new, he said, is real production use.

"It's really just in the last couple of months that we're seeing everyone from our biotech clients to really big banks and fintech starting to lean in this direction," Messer said. He added that early conversations were about cost, but now they are "a lot more about performance and control."

Victor Su-Ortiz of MiniMax agreed, with a caveat. Cost was "all the rage over the summer," he said, but the performance teams care about is how well a model does in one domain, such as legal or healthcare. More teams are fine-tuning and post-training open models for that purpose. Cost still matters, he noted, pointing to public usage data from OpenRouter and OpenCode, where fast, low-cost Flash models lead.

Think in systems, not models

Yunmo Koo of Friendli AI described the biggest change teams face in production as a move "from model-centric thinking to system-centric thinking." In experiments, a team checks a model card, plugs the model into a coding tool and sees whether it handles a task. In production, the question is whether the model can complete a complex agent pipeline reliably, within latency and cost budgets.

He offered an example: an endpoint that is 95% reliable drops to about 60% end-to-end reliability across a 10-step agent chain. His advice was to define success for the whole workflow, not for a single call.

Koo also warned that open weights do not automatically bring better economics or performance. Teams take on responsibility for model choice and inference configuration, and an inference provider can carry some of that load. Su-Ortiz added that cost per token is useful, but internal benchmarks on a team's own use cases matter most.

Long-horizon agents and the engine layer

Su-Ortiz said MiniMax's design shifted between its M2 and M3 series to give much more weight to long-horizon tasks. One-million-token context windows are now standard, he said, because agents prompt repeatedly across many turns. He described his own habit changing: he used to stay below 30% of the context window, but now goes up to 70 to 80%. To handle that load, MiniMax uses a sparse architecture and its own KV-cache optimizations.

Koo said labs now design for intelligence and efficiency together, using hybrid attention, cache compression and sparse indexing. But "theoretical improvement in the architecture does not always translate into real-world benefit." Linear attention, for example, does not use a standard KV cache, and without engine work tailored to how it manages state, it can slow inference down. Model labs and inference providers therefore need to work closely.

Safety, hybrid deployment and the human layer

Messer said open weights give enterprises more control and more responsibility. Requirements vary by client, but safety testing and red-teaming come up repeatedly, and teams must test both the model and the agent's behavior once it has access to it. Anaconda does this with Enkrypt AI, which it acquired this year.

He also described growing interest in local AI, such as running a model on a laptop for private work and later moving the session into a shared cloud environment. He was candid that hybrid is still more vision than settled pattern.

On what makes teams succeed, Messer pointed to people. Individuals often assume failures such as tool call errors are their own fault and do not share them. The most successful teams he sees share wins and failures across groups.

Advice for a first production agent

The panel closed with practical guidance for teams building a first production agent on open models:

• Start with a Flash model. Messer said they have come along fast and are efficient.
• Go specialized. Su-Ortiz suggested working with labs and providers on fine-tuning, post-training or on-prem needs for a specific domain.
• Build evals and observability into every step. Koo said this makes it quick to compare models and providers as things change.
• Define success for the full workflow. Track reliability, latency, throughput and cost end to end.
• Share what you learn. Tool-call errors and odd traces are team knowledge, not personal mistakes.

For teams weighing open weights, the message from the panel is that the model is only one part of the decision. Inference setup, evaluation on their own workloads and shared knowledge across the organization shape whether an agent works in production.