AWS adds three-tier Data Capture to SageMaker HyperPod

Amazon SageMaker HyperPod now includes Data Capture, a built-in capability that records live inference inputs and outputs to Amazon S3. AWS says the feature is designed to help teams detect drift, identify failure patterns and improve models using production interactions.
- Data Capture records live HyperPod inference data in Amazon S3.
- Three independent tiers cover payloads, load balancer metadata and model-pod data.
- AWS demonstrates Athena analysis, SQL labeling and fine-tuning in one cluster.
- Tier 3 can capture full payloads for debugging and training-data collection.
- AWS recommends encryption, sampling, lifecycle rules and PII redaction.
Three ways to capture inference data
HyperPod Data Capture provides three independent tiers, and teams can enable any combination in the same dataCapture block. Tier 1 is SageMaker endpoint capture, which stores full request and response payloads and is intended for compliance, Model Monitor and audit use cases.
Tier 2 captures Application Load Balancer access-log fields, including the request line, status codes, timings, client IP and user agent. AWS positions this tier for traffic and error-rate monitoring when teams only need metadata.
Tier 3 runs at the model pod. It stores full request and response payloads with event metadata and is intended for debugging, analytics and training-data collection. The feature uses an inference operator with NGINX, an OpenTelemetry collector and Amazon S3 uploader sidecars. AWS says Data Capture has minimal latency impact and zero impact on model availability.
A production-data flywheel on one cluster
AWS demonstrates the capability with a home-appliance support router using synthetic data. The walkthrough captures every request and response in Amazon S3, queries the records with Amazon Athena, labels them with a SQL policy, fine-tunes the router on the labeled data and redeploys the updated model alongside the original.
In the example, Athena surfaces a blind spot involving a new product. The fine-tuning job runs on a second GPU through the HyperPod training operator, while the inference operator serves the model on another node. The new model is then served by a second endpoint with capture still enabled. AWS says the example measures accuracy before and after fine-tuning on data that could not have existed before deployment.
The walkthrough uses Qwen2.5-3B-Instruct because AWS says it is small enough to fine-tune in about a minute and fits on a single NVIDIA L40S. The example requires a SageMaker HyperPod cluster orchestrated by Amazon Elastic Kubernetes Service, the inference and training operators, two available NVIDIA GPUs, kubectl and an Amazon S3 bucket in the same Region.
Controls for captured production payloads
Tier 1 and Tier 3 both store full payloads, so enabling both stores every record twice. AWS recommends Tier 1 for compliance and Model Monitor, Tier 2 when only traffic metadata is needed, and Tier 3 when teams intend to use payloads.
Because Tier 3 records can contain names, addresses and order numbers, AWS advises protecting the data before using it for training. Suggested controls include sampling, AWS Key Management Service encryption, Amazon S3 lifecycle rules and personally identifiable information redaction, for example with Amazon Comprehend.
In the deployment shown, Tier 3 is enabled with a 100% initial sampling percentage and captures both input and output, the defaults described by AWS. Teams can adjust the sampling percentage, buffer batch size, flush interval and maximum payload size.


