NVIDIA Brings Green Contexts to the CUDA Runtime API in CUDA 13.1
NVIDIA has detailed how green contexts, now accessible through the CUDA Runtime API as of CUDA 13.1, let a single application divide GPU execution resources such as SMs and workqueues between concurrent workloads. In an example on a Blackwell GPU with 148 SMs, NVIDIA says the approach cut a critical kernel's latency from 0.140 ms to 0.007 ms.
- Green contexts are now accessible through the CUDA Runtime API starting with CUDA 13.1; the Driver API has had them since CUDA 12.4.
- They let an application partition SMs and provision workqueues within a single process.
- NVIDIA says a critical kernel's latency dropped from 0.140 ms to 0.007 ms on a Blackwell GPU with 148 SMs.
- Streams are created from a green context handle instead of implicit thread-local device state.
- The feature is opt-in and additive, so existing applications can adopt it incrementally.
Why sharing a GPU inside one process is hard
NVIDIA's developer blog, written by Kyrylo Perelygin, Shelton Dsouza and Myrto Papadopoulou, starts from a common pattern: GPU applications increasingly run several independent components at once within a single process. Examples include a latency-sensitive operator next to a throughput-oriented background kernel, a data preprocessing stage next to model inference, or several stages of a processing workflow sharing one GPU.
According to NVIDIA, these components can interfere with each other unpredictably, and existing tools offer limited ability to partition resources. Traditional CUDA contexts were not built for this. NVIDIA describes them as heavyweight, with hardware context-switch overhead, and as reflecting assumptions from an earlier era when GPUs were smaller and applications typically ran one dominant workload.
Green contexts have been available in the Driver API since CUDA 12.4. Starting with CUDA 13.1 they are also exposed through the Runtime API, so an application can define, within its own process, where work runs and how execution resources are divided.
How green contexts partition resources
The main use is SM partitioning. An application assigns a subset of SMs to a green context, and work submitted through that context targets only those SMs. This allows multiple workloads to run at the same time without competing for the same compute units.
Green contexts can also provision workqueue resources. In the traditional model, independent stream-ordered workloads may map to the same underlying workqueues, which can introduce unintended serialization even when enough execution resources are free. By provisioning workqueues explicitly, applications can express the concurrency they expect and reduce false dependencies.
NVIDIA says green contexts are lightweight to create and destroy, and that doing so does not implicitly synchronize unrelated GPU work.
The programming model also changes. Historically, a Runtime application called cudaSetDevice(), created streams, and submitted work, with the stream's target inferred from the current device or context of the calling thread. With green contexts, the application creates a green context for a chosen set of resources and then creates streams from it. In the Runtime API the context is represented by the cudaExecutionContext_t type. The cudaGreenCtxCreate() call returns a handle that can be passed directly to calls such as cudaExecutionCtxStreamCreate(). In NVIDIA's words, "The code changes are minimal, but the mental model is clearer." Applications that want the full device can keep using the traditional model, and can obtain the device-wide context with cudaDeviceGetExecutionCtx() where an API needs an explicit handle.
A latency-critical kernel next to a bulk workload
NVIDIA's worked example addresses a familiar case: a small, latency-sensitive kernel sharing a device with a larger throughput-oriented worker. The blog cites communication and GEMM overlap in distributed training and inference, and latency-sensitive operators in AI sensor processing platforms such as NVIDIA Holoscan, as typical scenarios.
CUDA stream priority is a standard tool for such work, but NVIDIA notes it is not enough when bulk kernels fully occupy all of the GPU's SMs. Stream priorities cannot preempt a block already executing on an SM, so the critical kernel must wait for some blocks to drain before it gets SMs.
The sample code queries all SMs, carves out one critical group sized to the architecture's coscheduled alignment, and pairs each partition with its own workqueue-config resource so queue pressure is isolated too. It then creates a green context and a stream for each partition, with a high-priority stream for the critical kernel and a low-priority stream for the bulk work. NVIDIA says it tested three modes using the same critical and bulk workload. Mode A uses the green-context partition with a high-priority critical stream, and Mode B uses the default context with a high-priority critical stream and no partitioning. The provided text cuts off before the third mode and the full results.
The blog's summary states the headline result: on a Blackwell GPU with 148 SMs, critical kernel latency fell from 0.140 ms to 0.007 ms when the kernel had dedicated SMs and did not have to wait for bulk work to drain.
What it means for CUDA developers
NVIDIA stresses that green contexts are opt-in and additive. Existing applications keep working as they are, and the rest of the code can continue using the stream-based CUDA workflow, so teams can adopt green contexts step by step where they need finer control over concurrent workloads. For developers who mix latency-sensitive and bulk work on one GPU, the Runtime API access in CUDA 13.1 offers a more explicit way to isolate them. The company points readers to the CUDA Programming Guide section on green contexts and the CUDA Driver API reference for low-level resource partitioning.
NVIDIA Developer: Control How Your GPU Shares Work with Green Contexts
