NVIDIA Dynamo adds session IDs so inference servers can follow AI agents

Why agent traffic strains request-level serving

The post argues that agentic workloads have changed the traffic an inference server sees. A coding session opens with a large prefill, often tens of thousands of tokens of system prompt, tool definitions and user-specific guidance. Every later turn appends to a context that the session resends in full. One task can fan out into dozens of such calls, with short and long lived subagents running in parallel.

Most of a session's wall-clock time passes between tool calls, while its context sits resident in the KV cache and nothing generates. According to the post, serving this well comes down to how much of that context a system keeps resident and reuses, and how many sessions it can hold at once while still meeting latency targets.

The authors say most open-source serving stacks are still built for single-turn chat. They route, admit and cache each request on its own, with no notion of the harness or the session that links it to earlier turns.

One identifier for the whole chain of calls

A Dynamo session ID is a stable identifier for a single agentic chain of reasoning and tool calls. Every LLM request in a trajectory shares the same session_id, and child sessions usually carry a parent_session_id, so traces and replay tools can connect agents to the subagents or subtasks they spawn. The post stresses that every optimization it describes is an opt-in consumer of this information.

Dynamo reads identity headers that popular coding agents already send and maps them to an internal agent_context. According to the post, Claude Code, Codex and OpenCode work without configuration:

• Claude Code: x-claude-code-session-id for the root, x-claude-code-agent-id for child agents, and x-claude-code-parent-agent-id as the parent header.
• Codex: thread-id, with x-codex-parent-thread-id as the parent header.
• OpenCode: x-opencode-session-id, with x-opencode-parent-session-id as the parent header.

For harnesses that do not send a compatible header, the agent-plugins repo offers three integrations that translate native session identity into x-dynamo-session-id: a Pi provider, a Hermes middleware and an OpenClaw provider. Custom harnesses can opt in with one header, X-Dynamo-Session-ID, and add X-Dynamo-Parent-Session-ID if they support subagents. The post's example is a curl call to a chat completions endpoint with the header value research-run-42:researcher, and it says no proprietary API is needed.

Traces and replay without storing prompts

The authors describe a gap they call harness inference codesign. Agent observability tools capture what happens at the harness level, but they do not show how the inference stack performed underneath each step. Dynamo's answer is a lightweight trace keyed by the session ID, which can sit alongside other trace formats such as Harbor's ATIF or Letta's Trajectory format.

Tracing is switched on with DYN_REQUEST_TRACE=1. By default Dynamo does not store prompt, response or tool-call content, though these can be enabled. After each response stream finishes, it emits a request_end record with the session fields, output token count, finish reason, KV cache metrics, tool-call names and a replay block holding input length and per-block sequence hashes, similar to the Mooncake trace format.

Because a trace records the workload the agent sent rather than the agent's decisions, one capture can be replayed as often as needed without rerunning the model, the tools or external APIs. Sequence hashes become synthetic block IDs that preserve prompt-prefix sharing across turns and sessions without exposing original tokens. Offline, AISimulate runs the request graph against Dynamo's simulated scheduler, router and KV cache, so teams can compare worker counts, topologies, routing policies and cache capacity without spending GPU time. Live, AIPerf replays the same graph against a real Dynamo endpoint on real GPUs.

The team says it has used captured traces as backtesting data for its own deployments, and that the data also feeds internal autoresearch harnesses in which AI generates routing strategies and researches proactive KV cache offloading.

Scheduling and cache features built on the ID

The post lists further pieces that rely on the session ID. Session-aware scheduling lets routing and admission control account for session working sets and apply backpressure at tool boundaries to reduce cache thrashing. An experimental shared-pool indexer lets the router count reusable KV in external stores when placing requests. A Session-Prefix Indexer and a proposed KvHint interface are meant to support session-aware cache policies across memory tiers, with Dynamo supplying policy intent and the inference engines controlling execution.

What it means for users

For teams running coding agents on their own infrastructure, the practical point is low setup cost: the three named agents need no configuration, and other harnesses can opt in with a single header. The authors say they are working with harness providers to make this the default everywhere, and that Dynamo's CI tracks each first-class harness as it changes.

The portion of the post available here does not include benchmark figures for the scheduling or cache features, so their effect on latency or capacity is not quantified. What it does establish is the foundation: a shared session identifier that lets serving infrastructure reason about agent programs instead of isolated requests.