DigitalOcean Simulations tests AI agents across whole conversations, not single replies

Why single-response tests fall short

DigitalOcean illustrates the problem with a coding agent asked which Python version to upgrade to from 3.11. The agent picks one from memory, cites a few changes from the release notes and calls the upgrade straightforward. According to DigitalOcean, a single-turn evaluation asking whether this is a reasonable recommendation would likely pass it.

An experienced human engineer, the company says, works through the question over several exchanges. They check which versions are currently supported, lay out the options and let the user choose. Then they work out what breaks between the starting point and the target, including the releases in between, and check the code for it. None of that is visible in a single prompt and response. Simulations is built to close that gap.

The four components and key concepts

DigitalOcean describes a simulation as a staged conversation with a grader in the back of the room. Four components make it work:

• The candidate: what is being tested. This can be a managed agent on Harness Runtime, a model, a Dedicated Inference deployment, or an Inference Router that the simulation wraps in an agent.
• The scenario: a situation for the user simulator to act out, including what the user wants, who they are, what the agent should get done and how many turns to allow.
• The user simulator: a model that plays the user one turn at a time.
• The judge: a model that scores the finished conversation against the chosen metrics.

A turn is one exchange of a user message and the agent's reply, including tool calls. An episode is one scenario played once against the candidate, until the stopping criteria are met or the turn limit is reached. A trajectory is the record of an episode, and it is what the judge scores. A simulation run is a group of episodes covering every scenario in a scenario set, played as many times as requested.

The agent runs in an isolated Harness Runtime sandbox with its production configuration and Action Gateway tools. Because the judge sees every message and tool call at every turn, DigitalOcean says it can distinguish a reasoning failure from a tool-use failure and catch an agent that forgets earlier turns.

The Python upgrade example

The worked example uses a coding agent on Harness Runtime that maintains a small Python service called orders-api, which runs on Python 3.11. The agent can fetch web pages and search the repository. A good conversation takes three turns, each depending on the one before:

• The user asks which version to upgrade to. The agent fetches the Python release status page, returns the supported versions and how long each is supported, and asks what matters to the user.
• The user picks a version and asks what will break. The agent fetches the release notes for every version between 3.11 and the chosen one, not just the target, and lists what was removed.
• The user asks whether their code is affected. The agent searches the repository for the removed modules and suggests a replacement for each one it finds.

DigitalOcean notes that skipping the fetch in turn 1 may leave the options out of date, and forgetting 3.11 in turn 2 means the list misses everything removed in the releases in between.

How scenarios are written

A scenario is one JSON object in a scenario set, stored as JSONL with one scenario per line. There are three ways to create them: Scenario Builder, where a generator model writes a set from a sentence or two of description; export from production traces with Signals, which turns flagged Harness Runtime conversations into a replayable set; or writing them by hand and uploading the file.

Only the description field is required. Optional fields include a user persona, stopping criteria and a maximum number of turns. Stopping criteria are written as things the agent does, such as "The agent fetches..." or "The agent lists...". The user simulator keeps the conversation going until the agent meets every criterion, then ends it. If the criteria are left empty, the simulator decides when to stop based on the description, and otherwise the conversation ends at max_turns. An agent that never meets the criteria runs into that limit, which DigitalOcean calls a signal in itself.

The expected_outcome field is only needed for the Agent Policy Adherence metric, which checks whether the agent followed a rule or reached an end state. Other conversational metrics, including User goal completion, do not need it.

Because the versions come from a live web fetch, the scenario cannot script the user's choice in advance. For the same reason, the stopping criteria are written relative to the chosen version rather than naming one. Once a simulation run uses a set, the set locks, which keeps past runs readable.

What it means for teams

For teams building agents, Simulations offers a way to compare a baseline against a candidate. According to DigitalOcean, the same scenarios can be run against versions of an agent or two models to show what improved and what regressed. Every step is available through doctl evaluation and the DigitalOcean API. The source text provided is cut off partway through the scenario section, so details of the remaining stages, including scoring and comparison, are not covered here.