AWS brings managed NVIDIA Dynamo inference to SageMaker HyperPod

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, combining the open source framework's distributed inference capabilities with HyperPod's infrastructure management. AWS says the integration is designed to help serve large language models across multiple nodes and separate prompt processing from token generation.
- SageMaker HyperPod Inference now supports NVIDIA Dynamo.
- Dynamo enables multi-node, aggregated and disaggregated LLM serving.
- HyperPod manages GPU scheduling, networking, health monitoring and node replacement.
- AWS reports 2.05 times the highest aggregated throughput for one Qwen3 benchmark.
- Users can deploy through a DynamoGraphDeployment or InferenceEndpointConfig.
Managing large models and mixed workloads
AWS says serving large language models efficiently requires enough GPU memory and careful coordination between prefill, which processes prompts, and decode, which generates tokens. The company gives GLM-5.3 as an example: its FP8 weights exceed the 640 GB of GPU memory available on a single ml.p5.48xlarge instance, before the memory needed for KV cache and runtime overhead is included.
HyperPod Inference's NVIDIA Dynamo support enables multi-node deployment, disaggregated prefill and decode, and KV-aware routing. Dynamo works with inference engines including vLLM, SGLang and TensorRT-LLM. In disaggregated serving, separate prefill and decode worker pools can be scaled and tuned independently, while KV cache is transferred between them over Elastic Fabric Adapter networking.
Dynamo also supports aggregated serving, where prefill and decode run on the same worker. In both arrangements, workers can run on one instance or span multiple instances through tensor and pipeline parallelism. Its KV-aware router considers reusable prefix KV cache and active worker load. AWS says prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness can help avoid overloading workers.
What HyperPod manages
The HyperPod Inference add-on installs the Dynamo custom resource definitions and the components needed to deploy and schedule the serving system. The HyperPod Inference Operator manages endpoint deployment through an InferenceEndpointConfig resource, while the Dynamo operator manages resources in a DynamoGraphDeployment, including frontends, model workers, replica counts and resource requirements.
Grove coordinates related groups of pods, including multi-node workers and their scaling relationships. Kubernetes AI Scheduler places pods on GPU resources using gang scheduling, so the pods required for a multi-node worker are scheduled together. AWS Deep Learning Containers provide the Dynamo runtime, and the add-on manages integrated component versions.
HyperPod also provides node, GPU and EFA health monitoring, manages node replacement and works with the orchestration components to recreate affected workers. Deployments configured through InferenceEndpointConfig with dynamoSpec can be invoked through SageMaker endpoints.
Two deployment paths and benchmark results
Existing Dynamo users can bring their own DynamoGraphDeployment and continue using upstream documentation, recipes and manifests. Teams using HyperPod Inference can instead configure an InferenceEndpointConfig with dynamoSpec, using the configuration format and OpenAI-compatible interface they already know. Both paths support single-instance and multi-node deployments. vLLM is the default inference engine, with SGLang and TensorRT-LLM also available.
In an AWS deployment example for GLM-5.3, aggregated serving runs one model-worker replica across two ml.p5.48xlarge instances, with two frontend replicas. A disaggregated example uses one prefill replica and two decode replicas, each spanning two instances, for a total of six instances.
AWS benchmarked Dynamo on HyperPod with two models. For Qwen3-235B-A22B-FP8, using TensorRT-LLM on two p5en.48xlarge instances with 4,000 input tokens and 200 output tokens, disaggregated serving reached 207 output tokens per second per GPU, compared with 101 for the highest measured aggregated result. AWS reports this as 2.05 times the aggregated throughput, with the tested points meeting its targets for P90 time to first token and median generation speed.
In a separate test using Qwen3.5-397B-A17B-FP8 with vLLM, a two-node configuration delivered 126.4 output tokens per second per GPU, retaining 99.1 percent of single-node throughput under an agentic workload with high prefix reuse. The result illustrates the multi-node serving approach described by AWS for models and deployments that require more than one instance.


