Why agentic AI systems can produce different results each time

A spectrum from predictable to adaptive

The post was developed by the Exploration Workstream of the Generative AI Commons, a Working Group in LF AI & Data. Its purpose is to investigate developments beyond what is currently mainstream and inform the LF AI & Data community about subjects it may consider for future work.

It describes agentic AI systems as applications that can perform complex multi-step problem-solving, access tools in real time and use memory-based context. Their behavior can range from completely deterministic to highly nondeterministic.

Deterministic behavior is preferred when an autonomous process needs to be controlled and reproduced. Nondeterministic behavior can instead provide flexibility or faster execution when a problem is too large for a practical deterministic solution. For example, a real-time AI chatbot may give slightly different answers to the same question. That adaptability can suit product recommendations, user questions and content summarization, while healthcare scheduling, legal disclosures and financial disclosures require highly deterministic and reproducible workflows.

Why model inference can change an answer

One source of variation is the underlying language model and its inference process. Language model output involves sampling, which turns predictions into a probability distribution and selects the next token. Sampling techniques can introduce randomness to reduce repetition during generation.

The post also points to floating point non-associativity in digital computing. Although forward-pass actions in most language models are deterministic, token generation in many open source inference engines can depend on the batch sizes used during forward passes. As a result, even theoretically deterministic sampling settings do not necessarily make large language model APIs deterministic in practice when the underlying kernels are not batch invariant.

Batch-invariant kernels are available for various open source inference engines, according to the post, but they can increase inference latency. That tradeoff may explain why deterministic behavior is not always the default.

Changing tools and multi-agent context

Agentic components can also evolve over time. Coding assistants may receive newer versions of their underlying models, while agents can gain access to updated tools, memory capabilities or guardrails. Versioned rollouts of Model Context Protocol servers can change the context and responses used by enterprise agents, even when those changes are not visible to end users.

Multi-agent systems add another layer of complexity. They coordinate specialized components, with context engineering determining what each subagent receives and how responses are combined. The order of subagent execution, parallelization strategies and context synthesis can all affect the final output.

A temporary network delay affecting one subagent may change the context passed to the next inference step. The post says this can produce different application behavior even when tool responses themselves remain unchanged.

What this means for AI application design

The LF AI & Data Foundation argues that reproducibility requires deeper evaluation across the layers of an agentic application, along with adaptable system configurations. The issue therefore extends beyond choosing sampling parameters. Teams also need to consider inference kernels, model and tool updates, memory, guardrails, orchestration and context design.

For applications where outputs must be controlled and reproduced, these dependencies represent engineering considerations that need to be addressed throughout the agentic execution environment. For less regulated uses, variation may be an intended feature that supports adaptive responses.