AWS adds GPU diagnostics and observability to SageMaker HyperPod

AWS has introduced enhanced infrastructure observability in Amazon SageMaker HyperPod, bringing GPU diagnostics and profiling together for distributed training workloads running on Amazon EKS and Slurm. The capability is designed to help teams identify unhealthy accelerators, network problems and other causes of reduced training throughput.
- HyperPod now compares GPUs to identify outliers in training jobs.
- Static rules detect GPU faults, EFA degradation, throttling and overheating.
- Default metrics are collected every 30 seconds.
- CloudWatch is the new default metric store, with Prometheus still supported.
- The capability is available as an EKS add-on or Slurm extension.
GPU outliers and common failure patterns
Enhanced observability uses Amazon CloudWatch's DetectGroupOutliers API to compare GPUs within a training job. It evaluates utilization, memory usage, temperature, power draw and SM clock frequency, then flags GPUs that significantly deviate from their peers.
A diagnostic summary also applies static rules to known infrastructure problems. These include GPU XID faults, ECC memory errors, accelerator failures, EFA throughput degradation, packet drops, retransmissions, clock throttling and overheating. Each detected issue identifies the affected job, cluster, node, accelerator and rank.
AWS says the capabilities work with training jobs orchestrated by both Amazon EKS and Slurm. This is intended to help teams isolate a single problematic accelerator across large distributed jobs instead of searching through separate logs and dashboards.
Default and advanced infrastructure metrics
HyperPod collects telemetry at two levels. Default metrics are enabled automatically and cover cluster health, CPU, disk and operating system usage, GPU utilization and temperature, as well as per-job resource utilization. They are collected every 30 seconds and sent to the selected metric store, with approximately 1, 2% CPU overhead and 250, 300 MB of memory.
Advanced metrics are optional. They include the full node exporter suite, NVIDIA GPU DCGM and AWS Neuron metrics, and Elastic Fabric Adapter metrics covering throughput, drops, retransmissions and errors. AWS says these options increase the amount of telemetry sent to the metric store, which has a cost implication. The company describes them as particularly useful for long-running training jobs on large clusters where problems can emerge over time.
Amazon CloudWatch is now the default metric store, replacing Amazon Managed Service for Prometheus as the previously supported option. Teams can still choose Amazon Managed Service for Prometheus, and CloudWatch provides monitoring through CloudWatch SageMaker Insights in the AWS Management Console.
How HyperPod customers can enable it
For EKS-orchestrated clusters, enhanced observability is delivered as an add-on. For Slurm-orchestrated clusters, it is provided as an extension. AWS Identity and Access Management controls who can install or change the capability.
In the HyperPod custom setup workflow, observability is enabled by default, while customers can select advanced metric categories, configure logging, choose CloudWatch or Amazon Managed Service for Prometheus, and set up a Grafana workspace. In Quick setup, observability is enabled by default and cannot be unchecked.
AWS also provides console, API and AWS Command Line Interface options for installing the EKS add-on. Customers can begin with basic metrics and later enable categories such as advanced EFA metrics. The company says metric categories and profiling capabilities can be enabled or disabled at any time.
Why the update matters for distributed training
Distributed model training can lose significant GPU time when one accelerator, network link or host-side process degrades. AWS says HyperPod's combined metrics and diagnostics are intended to reduce the time spent determining whether a slowdown came from a GPU fault, thermal throttling, network degradation or another infrastructure issue.
For users running HyperPod training jobs, the update provides a more direct route from a performance problem to an affected job, node or accelerator. It also gives teams a choice between CloudWatch and Prometheus-based monitoring, while leaving deeper telemetry as an opt-in setting for workloads that need it.


