Datadog Adds Online Experiments to Compare AI Agent Changes in Production

Why agent changes are hard to judge

According to Datadog, improving an AI agent often means changing its prompts, models, retrieval methods or tools. Judging whether a change really helps is difficult. A revised prompt may give better answers on a handful of test cases but handle unexpected user requests poorly. A cheaper model may cut token costs while introducing tool-calling errors or longer execution times.

Datadog says teams need to understand how a change affects quality, reliability and cost under real usage before rolling it out. Online Experiments, available in Agent Observability, is built for that purpose. It combines agent evaluations with Datadog Experiments, and users can also inspect the underlying traces to see why behavior changed.

Offline versus online experiments

Datadog describes offline experiments as an integral part of agent development. Teams run agent versions against predefined datasets, compare outputs and catch known regressions before deployment. In Agent Observability, offline experiments cover both individual components and complete agent workflows.

The company points out the limits of this approach. Building comprehensive offline datasets takes time and product knowledge. Teams must choose representative requests, agree on what good behavior looks like and decide which checks reveal a regression. A model-based judge needs a rubric the team understands and a human to check its assessments. The work continues as the product changes.

Online experiments instead rely on online evaluations, which score agent behavior in production. Datadog says this lets teams catch patterns missing from offline datasets, and the findings can help build better offline datasets later.

The company gives the example of a support agent whose search tool returns long passages, leading to repeated searches. An offline test could check whether shorter excerpts still yield relevant citations, but might miss the full range of user requests or whether users actually resolve their issues.

How Online Experiments works

Teams keep the existing agent version as a control and test a changed version over the same period, with random assignment. According to Datadog, this separates the effect of the change from existing differences in usage, such as a new search tool looking better only because it receives easier questions.

Users can connect evaluators created in Agent Observability to experiment metrics, alongside operational measures they choose to collect. Experiments then calculates relative changes and confidence intervals. Datadog Feature Flags can manage the traffic split and adjust exposure without redeploying the application, so a team can start with a small share of users while keeping a control group.

Datadog stresses that measures should reflect the decision at hand. A revised tool might improve evidence quality but add searches, and a cheaper model might need more attempts to finish a task. Comparing evaluation scores with duration, cost and errors exposes those trade-offs.

Datadog's own test of a model change

Datadog used the feature on its Insights agent, which investigates agent behavior using trace evidence. It compared OpenAI's GPT-5.6 Luna and GPT-6 Luna, asking whether the lower model prices of Luna 6 meant cheaper investigations and what the switch meant for quality and tool behavior.

Traffic was split evenly, with Luna 5.6 as the reference. The primary measure was investigation quality, scored from 1 to 5 by an LLM judge using the request, tool inputs and outputs, and the agent's conclusions. The rubric covered accuracy, completeness and how well conclusions followed from the evidence, including whether the agent acknowledged uncertainty or missing data. Duration, estimated cost, estimated token usage and workflow errors were collected from traces as separate measures.

Luna 6's estimated cost was 44.3% lower, with a confidence interval from 37.5% to 51.2% lower. Datadog says this supported its hypothesis that the model change could reduce investigation cost. Quality scores were close, at 3.84 for Luna 5.6 and 3.86 for Luna 6, although Datadog notes that close averages alone do not establish equivalence.

What it means for teams

For teams running AI agents, the feature offers a way to test prompt, model, retrieval or tool changes against real traffic before a full rollout, with quality and operating costs in one view. Datadog's own example shows a cost reduction measured alongside quality, rather than assumed.