AWS adds SOCI lazy loading to cut SageMaker AI job start times from minutes to seconds

The problem: image downloads as billable overhead

SageMaker AI training and processing jobs run inside container images stored in Amazon Elastic Container Registry (Amazon ECR). Before a job can run any user code, the service downloads the entire image to the instance, then decompresses and writes every layer to disk.

For small images this takes seconds. AWS notes that deep learning images routinely exceed 10 GB, and that the download then adds minutes to every job, which customers pay for as job time. According to AWS, the cost is felt most in hyperparameter tuning sweeps with hundreds of jobs, quick experiments and short processing jobs, where image download can dominate total runtime.

How SOCI indexing works

SOCI is an open source technology for lazy loading container images. Instead of pulling the whole image first, the runtime uses a SOCI index to fetch only the files needed to launch the container, then streams the rest in the background as the code accesses it.

The index is metadata that maps every file inside an image. It is stored in Amazon ECR as an OCI-compliant artifact linked to the original image. Because it is a separate artifact, the image digest and any existing signatures remain valid, according to AWS.

SageMaker AI checks automatically whether the specified image has an index:

• Index present: the image is lazy-loaded and the container starts before the full download completes.
• No index: a standard download, with no change in behavior.
• Index unreadable or lazy loading fails: the job falls back to a standard download and keeps running.

AWS says this fallback means adding an index never prevents a job from running, and that no changes are needed to the job definition, entry point or data channels.

What AWS measured

AWS compared standard downloads with SOCI lazy loading on real SageMaker AI training jobs. The setup used eight images between 1.8 and 35.4 GB compressed, across GPU and CPU instances including ml.g4dn, ml.g5, ml.g6, ml.g6e, ml.g7, ml.g7e, ml.p4d, ml.p5 and ml.m7i, in US West (Oregon) and US East (N. Virginia). All figures are medians of repeated runs.

• 1.8 GB image: 48 s standard, 3-11 s with lazy loading, about 8x faster.
• 13.2 GB image: 3 min 42 s standard, about 6 s with lazy loading, about 34x faster.
• 35.4 GB image: 7 min 17 s standard, 3-11 s with lazy loading, about 38x faster.

Lazy loading stayed at 3-11 seconds regardless of size. For the 13.2 GB image tested across 13 instance type and size combinations, standard download ranged from 3 min 16 s to 4 min 20 s, while lazy loading held at 6-8 s.

AWS also ran full training jobs: ResNet-50 image classification on Imagenette with PyTorch and TensorFlow DLCs, and full-parameter supervised fine-tuning of Qwen2.5-0.5B on alpaca-cleaned. From the second epoch on, time per epoch was the same within run-to-run variation. The first epoch took 3-16 s longer from the lazy-loaded image while files were fetched for the first time. Net, each job finished 1-6 min sooner, according to AWS.

Where the gains shrink, and how to adopt it

AWS cautions that results depend on the workload. Lazy loading helps most when the first training step needs only a fraction of the image. If an entry point reads most of the image before training, such as by scanning or preloading contents, the net improvement would be small. In AWS tests, torch.compile and a just-in-time built CUDA extension added 7-15 s to the first epoch, with a net saving still between 3 min 25 s and 3 min 56 s. A 6 GB model baked into the image added 2-3.5 min at load time, but the job still finished 2-2.5 mins sooner.

There are two ways to adopt the feature. AWS publishes SOCI-indexed variants of several Deep Learning Containers, marked by a -soci suffix on the image tag. These are available for PyTorch Training, TensorFlow Training, vLLM and SGLang, including arm64 variants for some frameworks. Users update the image tag in the job definition. Alternatively, they can create an index for their own custom image.

One caveat: an image built FROM a DLC base must have its own index, because the base image index does not carry forward to derived images.

What it means for teams

For teams running many short jobs, the saving is a fixed amount per job, so it adds up in tuning sweeps and quick experiments. Because the change is a tag update or an added index, and jobs fall back to a standard download when needed, AWS positions it as a low-risk option. Teams should test it on their own workloads, since the first-epoch cost depends on what their code touches at startup.