Datadog Agent Observability adds audio to traces for voice agents

Why text traces fall short for voice

Datadog argues that a voice agent can return the right words and still deliver a poor experience. It might leave a caller in silence, speak over them, or stop midway through a response. A transcript may look correct even when the audio the user heard was delayed, incomplete, or different from the text on screen.

Voice conversations also involve more than one request and one response. Audio streams, transcripts, model interactions and tool calls all contribute to each turn. Datadog says teams need to examine those signals together and in the order the user experienced them. Audio keeps details a transcript cannot, including pauses, overlapping speech, and the point at which a response was interrupted.

What the feature captures

For supported applications built with the OpenAI Realtime API, automatic instrumentation captures each turn's input and output audio, transcripts, token usage and tool calls. Datadog converts raw PCM16 and G.711 audio to WAV, so it can be played directly from the trace. Agent Observability also adds tracing support for applications built with Amazon Nova 2 Sonic in the Python and Node.js tracers.

From an affected turn, a team can listen to the exchange while inspecting its transcript and tool calls. This helps determine whether the agent chose the wrong action, returned unexpected content, or was interrupted before it finished speaking.

For latency, the instrumentation emits the ml_obs.trace.time_to_first_agent_audio distribution metric for OpenAI Realtime applications. It measures how long a user waits for the agent's audio response. With percentile aggregations enabled, teams can compare values such as p50, p95 and p99. When the metric shows an increase, the "Average time to first agent audio" facet in Agent Observability can narrow the span list to the turns with the slowest initial audio responses. An affected trace can then show whether the delay relates to a model interaction, a downstream action or another part of the conversation.

Because providers count audio and text tokens separately, voice agents can have different usage patterns than text-only agents. Agent Observability captures token usage from OpenAI Realtime interactions, so high-usage turns can be reviewed next to latency and other operational signals.

Interruptions and incomplete transcripts

Datadog notes that interruptions are part of natural conversation, but repeated ones can signal a problem. An agent might give answers that are too long or that begin with irrelevant information, or it might rely on voice activity detection that mistakes a pause for the end of the user's turn. When a user interrupts an OpenAI Realtime response, the corresponding turn can be opened to hear what the agent was saying at that moment.

Transcripts need the same context. According to Datadog, OpenAI Realtime input transcription runs separately from the model's processing of the audio, and an output transcript can be incomplete when a user interrupts a response. Comparing the transcript with the audio keeps an incomplete transcript from being treated as a full record of the interaction.

Setup and availability

Automatic multimodal instrumentation comes through the Agent Observability SDK. Python applications that use the OpenAI Realtime API with ddtrace 4.13.0 or later can capture per-turn audio, transcripts, token usage and tool calls without manually constructing audio spans. Datadog says ElevenLabs Realtime API support will be available soon with ddtrace 4.17.0.

Applications that already emit spans compliant with the OpenTelemetry GenAI semantic conventions (version 1.37 or later) can also send supported audio message parts to Agent Observability. The automatic OpenAI Realtime instrumentation, however, is provided by Datadog's own SDK. Details on audio formats, SDK versions and media size limits are in the multimodal instrumentation documentation.

New Datadog users can start with Agent Observability for free. According to the company, the free plan includes up to 40,000 LLM spans per month.

What it means for teams running voice agents

For teams operating voice agents, the change moves debugging away from piecing together separate transcripts, metrics and logs. Datadog says users can go from a latency signal or a reported problem to the specific turn and trace that produced it, and hear what the caller heard while checking the model, tools, latency and token usage behind it.