AWS Adds Usage-Based Metering for SageMaker AI Streaming

A billing model designed for live streams

Customers using streaming models on Amazon SageMaker AI continue to pay for the endpoint infrastructure hosting those models. The new capability addresses a separate problem: how model providers can charge for usage during a live, long-running stream.

Existing AWS Marketplace metering for SageMaker AI real-time request-response inference relies on a unit count in an HTTP response header. That approach works when the billable quantity is known as the response is returned. Bidirectional streaming sessions can last 30 seconds or 30 minutes, however, with usage accumulating until the session ends.

A response header cannot report changing usage throughout that session. AWS says usage-based metering allows the Marketplace software charge to follow reported consumption, rather than charging based on the time a model is available, including quiet periods.

A companion WebSocket carries metering data

The system adds a second WebSocket connection alongside the existing inference stream. The data connection carries inference requests and responses, while a metadata connection is dedicated to metering information.

Model servers opt in by returning the X-Amzn-SageMaker-Metadata-Stream-Supported: true header in the WebSocket upgrade response. SageMaker AI then opens the companion connection at /invocations-bidirectional-stream-metadata, unless the provider specifies a custom path.

Both connections use the same X-Amzn-SageMaker-Request-Id, allowing the model server to associate metering records with the active inference session. Records are sent as JSON text frames, with one metering record in each frame. AWS recommends sending them incrementally as usage accumulates, followed by a final record when the session ends.

The metadata stream allows one message per second per connection, and each message can be up to 512 bytes. Records use the inference.count dimension, while ConsumedUnits reports the quantity. For example, 12 minutes of metered audio would equal 720 seconds when a product charges per second of audio.

Customers and providers keep separate responsibilities

The provider defines what counts as metered usage, and the AWS Marketplace offer maps that unit to a price. An open connection lasting 12 minutes does not by itself establish that 12 minutes of audio are billable. The product's counting rules determine the reported quantity, while SageMaker AI endpoint infrastructure charges remain separate.

For customers using models for transcription, contact centers, voice agents or captioning, AWS says usage flows into the consolidated AWS bill. Customers can also use negotiated rates through private offers and do not need to manage a separate billing relationship with the model provider.

Model providers can configure their servers to report usage during streaming inference sessions. The SageMaker Marketplace Onboarding Agent Skill includes an opt-in inference metering path and starter templates that advertise metadata-stream support and create incremental ConsumedUnits frames. AWS advises providers to review and production-harden the code before submission.

Providers can choose whether inference continues if the metadata connection fails. The X-Amzn-SageMaker-Metadata-Stream-Required header defaults to false, allowing inference to continue in data-only mode. Setting it to true stops inference when the metadata stream becomes unavailable, prioritizing metering continuity over inference availability. AWS cautions that transport delivery does not guarantee malformed or dropped records will be billed, and recommends validating reported units against AWS Marketplace reports and buyer charges.