NVIDIA AI Cluster Runtime Packages Dynamo Into Validated Kubernetes Recipes

Why the runtime layer matters

NVIDIA Dynamo is a distributed inference framework that coordinates model serving across GPU workers. On Kubernetes, a model is described by a DynamoGraphDeployment, which specifies how vLLM, SGLang or NVIDIA TensorRT LLM should run in pods. NVIDIA says that graph is only the top of the stack: GPU drivers, resource allocation, scheduling, networking and monitoring all have to work together, and resolving those dependencies takes time away from testing model behavior, serving performance and application integration.

AICR is part of the NVIDIA DSX AI Factory Platform. Each recipe defines a configuration for a specific environment and workload, including pinned component versions, compatibility constraints and validation checks. The recipe also records configuration values, node selectors, tolerations, checksums and validation expectations to make deployment reproducible.

What an AICR Dynamo recipe includes

The recipe ships the dynamo-platform chart, pinned to version 1.4.2. It contains the Dynamo operator, Dynamo runtime images and the nvidia.com/v1beta1 DynamoGraphDeployment CustomResourceDefinition, which replaces the former v1alpha1 API version. Supporting components listed by NVIDIA include:

• NVIDIA GPU Operator for the driver and GPU software stack
• Node Feature Discovery for labeling and discovering GPU nodes
• DRA Driver for NVIDIA GPUs for Kubernetes-native Dynamic Resource Allocation
• KAI Scheduler for topology-aware placement and gang scheduling
• Grove for starting, placing and scaling multi-component serving graphs as coordinated groups
• NVSentinel and Nodewright for node health signals, where supported by the selected recipe
• Prometheus stack for cluster, accelerator and Dynamo service metrics
• agentgateway and Gateway API resources for Kubernetes-native ingress

With Dynamo 1.4, frontends, workers and optional Gateway API Inference Extension (GAIE) / Endpoint Picker Plugin (EPP) components discover each other through Kubernetes resources. AICR's default path therefore no longer uses NATS, and the dynamo-platform chart no longer installs it by default. NATS remains available as an opt-in.

Cache-aware routing and two paths

NVIDIA points to agentic workloads, which repeatedly extend long token prefixes such as system prompts, tool definitions, conversation history and tool results. If a request reaches a worker without that prefix in GPU memory, intermediate tokens are reprocessed instead of reused. Dynamo therefore routes requests to workers that already hold the relevant KV blocks. It can also separate prefill from decode, known as disaggregated serving, so operators can scale and tune each pool independently for a given request mix.

In Dynamo 1.4.2, the default AICR Kubernetes path uses TCP for requests and ZMQ for KV-cache events. Each vLLM worker emits raw KV-cache events through a local ZMQ publisher. Dynamo's worker runtime subscribes to that source, normalizes the events and republishes them onto the Dynamo event plane. Both routing paths use this KV-cache state.

The first path uses the Dynamo Frontend router. Running with DYN_ROUTER_MODE=kv, the Frontend tokenizes each request, scores eligible workers by prefix overlap and current load, and dispatches the request over TCP by default. NVIDIA describes it as suited to a model-specific endpoint owned by an application team, as the smallest topology and the most production-proven serving option. The second path uses a Kubernetes Gateway API implementation for ingress and Dynamo EPP for endpoint selection, and integrates with shared Kubernetes ingress. AICR ships and validates the runtime for either topology.

What it means for platform and application teams

According to NVIDIA, AICR separates runtime setup from application deployment in a five-step flow: snapshot the cluster, generate a Dynamo inference recipe, render a deployment bundle, deploy the runtime with Helmfile or GitOps, and validate the installed runtime before installing application workloads.

For platform teams, this offers a repeatable way to deploy and check the runtime. For developers, NVIDIA says it provides a consistent foundation for serving models and testing applications. NVIDIA points readers to the AICR and Dynamo repositories and to the DSX AI Factory Slack community for further discussion and workshops.