AWS SageMaker adds GPU partitioning for more efficient model hosting

Sharing one GPU across multiple models

Before this support, the smallest GPU resource available when deploying a model to a GPU endpoint was one whole accelerator. AWS gives the example of an ml.p4d.24xlarge instance with eight A100 GPUs. Under the previous model, it could host eight models, even when individual models needed substantially less than the available GPU memory.

GPU partitioning uses NVIDIA Multi-Instance GPU, or MIG, to divide a physical accelerator into independent partitions. Each partition receives a fixed share of compute and a dedicated block of memory, with isolation enforced in hardware. This lets customers place different models on separate partitions and use a fraction of a newer accelerator rather than choosing between a whole older GPU and a whole newer one.

NVIDIA MIG can split one GPU into as many as seven instances, depending on the accelerator. AWS says the partitions provide predictable latency, fault isolation, right-sized capacity, no model cold starts caused by eviction, and hardware-enforced tenant isolation. Different slice sizes can also be combined on one device.

A managed alternative to existing workarounds

AWS describes three approaches customers have used when they have more models than accelerators. They can run fewer, larger endpoints and batch requests, although that works only when models are interchangeable. They can use multi-model endpoints, which load and unload models on demand but do not provide device isolation and can introduce a load penalty on first use. Or they can run self-managed Kubernetes, where MIG requires work such as driver versioning, partition configuration, device plugins and fleet consistency.

With GPU partitioning on Amazon SageMaker AI, customers select a partition profile for a model, while SageMaker AI handles GPU configuration, placement and scaling. The feature works with endpoints that use inference components, which allow several models to run on one endpoint with separately assigned compute. Single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned.

Salesforce helped define requirements and validate the initial implementation. Richard Chang, a Salesforce software engineering architect, said MIG could help the company “consolidate workloads safely, significantly improve GPU utilization, reduce inference costs, and maintain predictable performance and strong isolation.”

How configuration and capacity work

The feature adds two optional fields to existing API calls. AcceleratorPartitionLayouts is specified on a production variant in CreateEndpointConfig and defines how each accelerator is divided. AcceleratorPartitionType is specified on an inference component and selects the partition used by its model. AWS says there are no new API operations, resource types or IAM actions, and endpoints that omit both fields behave as they do today.

In AWS’s example, an A100 is divided into mig-4g.20gb, mig-2g.10gb and mig-1g.5gb partitions. Because each A100 has seven compute slices, the layout uses all seven. On an ml.p4d.24xlarge instance, that produces 24 partitions, with eight of each type.

Each inference component copy occupies one partition of its requested type. Increasing CopyCount can place additional copies on free partitions, which AWS says can help a small model handle higher concurrency. Invocation remains unchanged because InvokeEndpoint addresses the model by inference component name.

Partition layouts are selected when an endpoint is created, and a model that needs more than one GPU still requires whole devices. AWS measured roughly 17 seconds of additional provisioning time for carving accelerators, separate from normal instance launch and container image pulling. For users, the change means more granular GPU allocation and managed hardware isolation, while the source's stated benefits include improved utilization, capacity matching and clearer cost allocation.