DigitalOcean Signals flags where AI agents loop, fail or frustrate users

The problem: green dashboards, poor sessions

DigitalOcean argues that standard metrics miss how an agent actually performed. In its example, a user spends ten minutes retrying the same failing command with slightly different arguments, yet the infrastructure sees nothing wrong: the run started, the run finished and the tokens were billed. Because agents are non-deterministic, the same prompt can go well one day and badly the next, according to the company.

Reading transcripts by hand does not scale, the company says, and asking another model to grade every conversation can become as expensive as running the agent itself. Signals is meant to sit between those two options.

What Signals detects

Signals covers 25 signal types in seven categories, grouped into three layers. DigitalOcean says the taxonomy comes from research its team published earlier this year on arXiv about triaging agent trajectories.

• Interaction signals describe how the user responds: misalignment, stagnation, disengagement and satisfaction.
• Execution signals show whether tool calls succeed. Failures include invalid arguments, bad queries and credential misuse. Loops include plain retries, parameter drift and oscillation between two actions.
• Environment signals cover external API errors, context overflow, malformed responses, network failures, rate limits and timeouts.

Not every signal indicates a problem. Satisfaction signals mark sessions where the user confirmed the answer, said thanks or reached their goal. Each signal carries the run, the step index, a confidence score and a short snippet.

Detection involves no model call. According to DigitalOcean, it is deterministic and inexpensive, so the cost does not climb with traffic and the same conversation yields the same signals every time.

Triage, not grading

DigitalOcean describes Signals as a first-pass triage tool that decides which conversations deserve a reviewer or a more expensive LLM judge. For each session it writes a report that rates quality from Excellent to Severe, lists the detected signals and flags the session if it warrants a closer look.

In a controlled study on τ-bench, a benchmark for tool-using agents, annotators found 82% of the sessions Signals flagged to be informative, meaning they showed a real failure or a good example. For randomly chosen sessions the figure was 54%. The company says this means reviewers find about 1.5 times as many useful sessions for the same budget, and cautions that production results will vary.

Flagged sessions can be exported as datasets that feed Evaluations and Simulations, so production cases can be used to evaluate and improve agents.

Privacy and setup

Signals is off by default and is enabled per environment. The consent step is recorded and reversible, and it only analyzes conversations that begin after it is switched on. DigitalOcean says conversation content stays within the customer's team boundary and is never used for cross-customer model training. Exports redact personal information from free text, and a deletion request removes collected data.

Users need doctl 1.181.0 or later and a token with the signals:query and signals:update scopes. Accounts in a free trial cannot use Signals during the preview. Consent can be given in the Control Panel on the Harness Runtime page or with the command doctl signals consent set --agent-id <agent-id> --enabled=true. Inference Engine signals are available only through the API and doctl for now, with a Console experience to follow.

Signals typically appear within 30 minutes of a session going idle, and metrics in the Signals view can take up to an hour to reflect recent activity. Sessions are reanalyzed when closed or destroyed.

What it means for teams running agents

For teams running agents on DigitalOcean, Signals offers a way to review a whole agent at a glance and open only flagged sessions, without adding instrumentation to agent code. DigitalOcean says there is no cost for using Signals during Public Preview for both Managed Agents and Serverless Inference. Pricing after the preview is not stated in the source.