AI21 Labs Replaces Slack-Style GPU Haggling With Kueue on a 10,000-GPU GKE Cluster

From a chat channel to automated scheduling

AI21 says its GPU sharing started out as a conversation. Teams used an internal messaging channel called #gpu-resources: when someone needed a GPU type that was fully used, they asked in the channel and hoped another team would free capacity.

That approach worked while the cluster had headroom. Once utilization stayed pinned near 100%, which AI21 describes as the goal for an expensive reserved GPU fleet, every request turned into what the company calls a zero-sum eviction conversation. AI21 lists the resulting problems as wasted engineering time on triage and negotiation, unfair queueing, idle capacity in some pockets while others starved, and reliance on ad hoc "magic commands" to free up capacity.

The case study covers one shared GKE cluster on Google Cloud. AI21 runs it as a single pool rather than one cluster per team on purpose, because Kubernetes makes it easy to create many clusters but hard to share capacity between them.

Four design principles and why vanilla Kubernetes fell short

AI21 defined four pillars for an ideal GPU management system:

• Fairness: scheduling decisions follow account usage rather than first-come-first-served or loudest-voice-wins.
• Order: queuing reflects business priorities, not just timestamps.
• Efficiency: developers no longer have to manually delete idle or non-critical workloads.
• Technical integrity: multi-pod (gang) jobs do not start until all resources are ready.

The company first checked what Kubernetes ships with. PriorityClass works at the pod level and has no gang-size awareness, so entire lower-priority runs can be preempted even when only one of several GPUs is needed. ResourceQuota allows no borrowing across teams when capacity is idle, and over-quota pods are rejected at admission rather than queued. LimitRange acts as a guardrail, not a scheduler. Job and Indexed Job lack an all-or-nothing approach, so if three of four pods schedule and the fourth cannot fit, the first three still run and burn GPUs.

Because GKE does not ship with a batch scheduler, AI21 evaluated Apache YuniKorn, Volcano and Kueue. At that time, it says, Volcano had more features but did not integrate seamlessly with its environment, and YuniKorn did not cover all its use cases. Kueue won on simplicity and integration.

How Kueue maps to the requirements

AI21 describes Kueue as a job-level manager that decides when a job is admitted to start and when it should stop. It matched the four pillars as follows, according to the company:

• Fairness: Preemption Fair Sharing balances cluster usage by preempting workloads, while Admission Fair Sharing (AFS) admits workloads first from teams that historically used less.
• Order: Workload Priority Classes control both queue order and preemption order.
• Efficiency: the ClusterQueue gives admins fine-grained control over preemption and integrates Fair Sharing and Workload Priority Classes, and Kueue handles workload cleanup automatically.
• Technical integrity: all-or-nothing semantics work with Indexed Jobs, so multi-pod gangs are admitted only when every pod can run.

The cluster handles three machine capacity types (reserved, spot and Dynamic Workload Scheduler flex-start mode) and three kinds of workload: debug pods, training jobs and inference models used by Reinforcement Learning jobs. After talking with the Algorithm Developer Leads who manage cluster usage, AI21 defined pools including per-team guaranteed quota that is never preempted, shared guaranteed pools for single-node and multi-node jobs, an opportunistic pool for best-effort work such as debug pods, and an on-demand pool for other cases. Per AI21, the opportunistic pool is the lever that keeps utilization high.

A collaboration that fed back into Kueue

AI21 says adopting Kueue grew into a close collaboration with the Google team behind it, with its production needs feeding into Kueue's roadmap. This includes features such as Admission Fair Sharing, which the company says is now available to all users. The source text provided breaks off while describing the first design rollout, labeled v0.10, so further results and lessons are not covered here.

For teams running large training workloads on shared Kubernetes clusters, the case offers a concrete example of replacing ad hoc negotiation with policy-driven admission, prioritization and preemption.