DigitalOcean launches Signals and Simulations to catch agent failures

Signals: finding failures that look like successes

According to DigitalOcean, standard observability can show that a system was up and latency was fine, but it cannot show that an agent chose the wrong tool, used stale context, or left a frustrated user giving up. Signals is meant to close that gap. It watches production traffic on DigitalOcean Managed Agents and DigitalOcean Inference Engine and automatically highlights potential problems such as frustrated users, failing tools and infrastructure constraints.

Signals stitches conversation turns into sessions and classifies each one across three dimensions:

• Interaction signals track user sentiment, including corrections, escalations and users quitting.
• Execution signals track tool reliability, such as invalid arguments and retry loops.
• Environmental signals track infrastructure, including timeouts, rate limits and context overflow.

Trends appear in the Signals view on Harness Runtime and Serverless Inference, and users can drill from a trend into the underlying sessions.

Cheap triage instead of grading every session

DigitalOcean says raw trace volume can exceed 100 GB/s, which makes grading every session with an LLM judge prohibitively expensive. Signals instead runs an inexpensive first-pass triage to pick out sessions worth a closer look. The company describes the method in its paper Signals: Trajectory Sampling and Triage for Agentic Interactions. In a controlled study on the τ-bench benchmark, it reports an 82% informativeness rate and a 1.52x efficiency gain over random sampling.

Flagged sessions can be turned into evaluation scenarios in one click. Signals is opt-in and, according to DigitalOcean, is designed to redact PII before processing. Traces and trace-derived evaluations stay within the customer's DigitalOcean Team boundary and are never used for cross-customer model training. The company says it works best when the agent runs on Managed Agents and models use DigitalOcean Inference Engine.

Simulations: testing whole conversations

Simulations is a new capability of DigitalOcean Evaluations. DigitalOcean notes that single-response test cases miss failures that develop across turns, for example when a prompt tweak fixes one step and quietly breaks a later one. Simulations runs multi-turn scenarios that start from the agent's own configuration, so they only exercise goals and tools the agent supports.

When Signals is enabled, scenario generation is weighted toward areas where production traffic already shows friction and toward what the change is most likely to affect. Without production data, guided refinement builds a baseline set from the configuration. Users can also describe a scenario in plain language, such as "a refund request from an angry customer with missing order details."

A user simulator plays the persona turn by turn against the candidate, which runs in an isolated sandbox on Harness Runtime with the same configuration and Action Gateway tools used in production. Tools that write data or cost money can run as mocks. Each scenario plays several times, and an LLM judge scores every trajectory on Evaluations metrics such as user-goal completion, knowledge retention and tool-use accuracy, plus any custom metrics. Because each trajectory records every tool call, Simulations can distinguish a reasoning failure from a tool-use failure, and comparing baseline with candidate shows what improved and what regressed.

Supported harnesses and how to start

Simulations can validate changes to instructions and tool selection, as well as fine-tuned or RL-trained checkpoints, on any harness supported by Harness Runtime, including Claude Code, Codex and OpenCode. Third-party agents such as LangChain, Hermes and CrewAI are not supported in this preview. For comparing two models on fixed prompts, DigitalOcean says single-turn Evaluations remains the right tool.

To use Signals, users open Harness Runtime in the DigitalOcean Console, find Signals and enable it for one or more Managed Agents. Data processing can take up to an hour to begin. Flagged sessions can be exported to Evaluations as test scenarios, and Signals data can be downloaded to fine-tune models or used as reward and preference data for reinforcement learning.

To run a simulation, users go to Evaluations & Simulations, choose a scenario, and click Run Simulation. They then select the candidate, the user simulator model and the metrics, and set episodes per scenario and the turn limit. Scenarios worth keeping can be saved to a library as regression tests.

What it means for teams running agents

DigitalOcean frames the two tools as a loop: Signals shows what went wrong, Simulations tests the fix before it ships, and Signals keeps watching afterward. Because Harness Runtime and Action Gateway already emit events, the company says nothing extra needs to be instrumented for agents on Managed Agents. For teams already running agents on DigitalOcean, that means production problems can feed directly into pre-release tests and later regression checks.