AWS Adds Cross-Region Inference Routing to SageMaker AI

AWS has introduced cross-Region inference routing in Amazon SageMaker AI, giving applications one invocation path across model endpoints and inference components in multiple AWS Regions and accounts. The new inference router keeps requests local when possible and shifts new traffic when a local destination cannot serve reliably.
- AWS introduced cross-Region inference routing in Amazon SageMaker AI.
- The inference router prefers local destinations and shifts traffic when they cannot serve reliably.
- Routing targets can contain endpoints and inference components across Regions and accounts.
- The router retries eligible real-time failures once on another destination.
- Health states and routing decisions are available through Amazon CloudWatch and response data.
A managed path across Regions
Before this launch, customers running models on SageMaker AI managed endpoints had to build their own cross-Region routing logic. That meant detecting a struggling endpoint, deciding which failures were safe to retry, moving traffic back after recovery and managing access to endpoints in other accounts.
The inference router is a Regional SageMaker AI resource that applications invoke instead of calling a specific endpoint. Customers register the destinations that serve a model, including real-time endpoints, inference components or asynchronous endpoints. A routing target identifies one logical model, and a router can contain multiple routing targets that fail over independently.
Endpoints remain deployed as they are today, with their existing models, configurations, instance pools and autoscaling. The router does not create capacity, scale endpoints, deploy models or copy model artifacts and container images. Each destination must already be running in its Region.
How SageMaker AI chooses a destination
The router prefers destinations in its own Region. If multiple local destinations are healthy, it spreads requests across them evenly. Requests move to other Regions only when local destinations cannot serve reliably, and traffic returns gradually after recovery.
SageMaker AI calculates a health score from recent requests in each destination's own Region. Server errors, throttles and timeouts reduce the score, while platform errors such as a failed health check have the greatest effect. Errors caused by the request itself, such as invalid input, do not reduce the score. Model latency is not used as an input.
The service publishes one of three health states to Amazon CloudWatch: healthy, overloaded or degraded. An overloaded destination continues serving while some traffic moves away. A degraded destination receives most of its traffic elsewhere, while retaining a small share to detect recovery. AWS says traffic starts moving within about a minute of sustained errors, or longer for destinations with little traffic. Customers cannot set routing weights or a fallback order.
Retries, asynchronous requests and supported setups
For real-time requests, the router retries once on another destination when the first destination fails without running the model. AWS lists throttles, a 503 error, a model that is not ready, rejection by a cross-Region traffic cap and access denied by the destination as examples. It does not retry timeouts, 500 errors or model errors. Requests already in progress are not interrupted.
For asynchronous requests, the router tries each eligible destination once, starting with local destinations, until one accepts the request. Applications use InvokeInferenceRouterAsync with a routing target name and an Amazon Simple Storage Service input location.
AWS describes use cases including handling GPU capacity shortages, routing between endpoints owned by separate AWS accounts, serving several models from one router and placing a router behind an existing API or model gateway. Direct InvokeEndpoint calls continue to work, and real-time responses include the destination that served the request and the reason for the routing decision.


