Salesforce uses SageMaker AI routing to manage global model capacity

Salesforce has integrated Amazon SageMaker AI’s cross-Region inference routing into its Internal Gateway to handle fragmented model capacity across more than 30 global Regions. The setup can move eligible requests to Salesforce endpoints in other Regions and accounts while keeping residency-restricted traffic in its original Region.
- Salesforce serves models across 30+ global Regions and hundreds of AWS accounts.
- SageMaker AI’s router can direct requests to Salesforce endpoints in other Regions and accounts.
- Residency-restricted requests remain pinned to their in-Region endpoint.
- Routing responds to capacity and health signals, including throttles and timeouts.
- Salesforce logs the routed resource and routing reason for operational visibility.
Capacity fragmentation created a reliability problem
Salesforce’s Agentforce Foundation team serves models for products including Agentforce and the Salesforce Trust Layer. Its fleet spans 30+ global Regions and hundreds of AWS accounts, with capacity provisioned according to each customer’s traffic.
Because customer peak windows often overlap, provisioning for every possible combined peak would cost too much. Adding capacity Region by Region can also leave some Regions with idle instances while others run out of capacity. Customers may additionally onboard new business units at short notice, requiring Salesforce to expand its fleet quickly.
Before using SageMaker AI’s inference router, Salesforce’s Internal Gateway sent traffic that local endpoints could not handle to other Regions. That system used health checks, retries and Region-switching logic, but it lacked capacity signals from local and remote endpoints. Cross-Region requests also left the virtual private cloud through a NAT gateway and an internet gateway and traveled over public AWS networking.
How SageMaker AI’s inference router works
A cross-Region inference router is a Regional SageMaker AI resource in a customer’s account. An application invokes the router instead of an endpoint, using the same request body and content type. Each routing target represents one logical model and lists the endpoints or inference components that can serve it by their full Amazon Resource Names.
Destinations can be in another Region or AWS account. The endpoint owner grants access through a resource-based endpoint policy that allows a named router to invoke it. The router prefers its own Region and keeps requests there while the local destination serves successfully.
The router assesses destinations using recent request outcomes. Platform faults, model-container 5xx errors, throttles and timeouts reduce a destination’s health. As problems persist, it shifts a growing share of new requests to healthy destinations, typically starting within about a minute, although a destination with little traffic can take several minutes to react.
After recovery, traffic returns over several minutes, more slowly than it moved away. The router retries a request once on another destination when the original failure occurs without running the model, such as a throttle. Invalid-input errors and model latency do not trigger traffic movement. The router uses capacity that has already been deployed, while each endpoint’s autoscaling adds instances as traffic grows.
Salesforce keeps residency decisions in its gateway
Salesforce placed the router call where its direct InvokeEndpoint call had been in the Internal Gateway’s per-request egress path. The gateway still performs the data residency check before invoking the router. Requests with a residency requirement are pinned to an in-Region routing target whose only destination is the local endpoint, preventing failover across a boundary.
Other requests use the router with the same body, content type and accept type as before. Capacity awareness, health assessment and failover now sit in the routing layer rather than Salesforce’s application code. Salesforce also designed a cap on cross-Region traffic to reduce the risk that shifting traffic during an incident could overload another Region.
The gateway records InvokedRoutedResourceArn and RoutingReason alongside its request and trace IDs, and exposes them as application performance monitoring tags. The ARN shows the destination’s Region and account, while the routing reason identifies outcomes such as local-preferred, failover, capacity-available or only-healthy. Salesforce joins these fields with Amazon CloudWatch metrics when investigating traffic shifts.
What the setup means for model-serving operations
AWS says the arrangement gives Salesforce one invocation path across SageMaker AI endpoints in several Regions and accounts, using endpoints that Salesforce has already deployed. In the example described by AWS, Salesforce created a router in US West, Oregon, then tested access to an endpoint in US East, Ohio owned by a different account.
For users of Salesforce services, the design is intended to improve the team’s ability to absorb uneven demand without treating every regional capacity shortfall as an availability problem. At the same time, residency-tagged requests remain confined to their assigned Region. For Salesforce’s operators, the routing fields provide request-level visibility into where traffic ran and why, while the router’s own retry behavior avoids adding another retry layer in the gateway.


