Ai2 replaces priority scheduler with GPU time budgets to fund high-impact research

Demand far above supply

Ai2's AI Infrastructure team manages thousands of NVIDIA H100, B200 and B300 GPUs, arranged in clusters ranging in size from 88 to 1024 GPUs. The clusters are built for large-scale distributed training and serve about 150 internal researchers. Their work covers the full model flow of LLM and VLM training, robotics reinforcement learning (RL) simulation, and post-training for scientific agentic use cases.

As in many labs, demand exceeds supply. Based on submitted workloads, Ai2 says it has outstanding requests for 2-3x more GPUs than are available at any moment, so every available GPU hour has 2-3 research workloads competing for it.

The team frames its job as a pyramid of four metrics. Availability is the base, followed by occupancy, then impact, with utilization as the capstone. The post focuses on impact, defined as how often the most valuable workloads are chosen to receive resources.

What went wrong with priorities

The old scheduler was priority-based, and workloads could opt out of preemptability. Each team had a limit on concurrent GPUs for workloads protected from preemption, while preemptible workloads could exceed that limit on idle GPUs.

Ai2 describes several predictable problems:

• GPU squatting: users parked no-op workloads they could connect to when needed, because they could not launch debugging workloads with low enough latency to work on problems in real time.
• Priority inflation: eventually 100% of scheduled workloads used HIGH priority, which starved lower priority levels of GPU time.
• Support burden: because preemptability was optional, on-call engineers spent a majority of their ticket response time negotiating the organized shutdown of non-preemptable workloads on hosts with known maintenance problems.

Ai2 says it was slow to identify the root causes. Early efforts focused on tighter control of priorities and on assigning GPU monopolies to important projects. The team now sees the situation as a tragedy of the commons, with individuals competing over a scarce shared resource and producing a non-optimal global result. It also cites the 2011 Dominant Resource Fairness paper by Ghodsi et al., which recounts a search company whose users "would sprinkle their code with infinite loops to artificially inflate utilization levels."

Monopolies were too coarse, Ai2 says. Because teams are ready to run experiments at different times, dedicated GPUs sat idle while other teams waited for capacity.

Budgets and hierarchical fair-share

Instead of giving teams GPUs, Ai2 now allocates a portion of GPU time. The reasoning is that future demand depends on the results of novel experiments and cannot be forecast precisely, while priority across research efforts is a strategic question that can be debated in advance. Leadership is meant to think like investors, deciding how to fund each effort based on its likely impact before workloads exist.

Managers can proportionally allocate GPU time to the projects and researchers they are responsible for. In Ai2's example diagram, Project A1 has a 35% claim on total capacity regardless of how many other projects are queuing. Every request for GPU time must be funded by a budget, or it is not protected from preemption. A squatting workload, in Ai2's words, is spending team budget on nothing. The aim is to make gaming the scheduler more expensive than arguing honestly for a larger budget.

Allocation decisions within a project are made by a lead researcher, within a program by a principal investigator, and across programs by a lead program manager or the CEO.

A hierarchical fair-share scheduler manages actual occupancy. Ai2 notes the algorithm is not new, tracing it to the Hadoop Fair Scheduler in 2009 and to SLURM's Fair Tree and YARN's Fair Scheduler today. What is new, it says, are the inputs: the tree mirrors the research program structure, and weights are budgets set by managers rather than static quotas. The scheduler tracks occupancy over a sliding lookback window, 7 days by default, and ranks workloads from under-utilized allocations above those from over-utilized ones.

What it means for researchers

Ai2 expects every group to receive its allocated GPU time over a week-long range, as long as it keeps submitting workloads with enough demand. The post quotes Chris Clark: "The new scheduler makes it feel like we have an extra 30% compute." He says bursty workloads can now exceed the allocation limit later and still be scheduled quickly and without preemption, which lets the team reclaim compute that the old scheduler effectively lost.

The shift also moves a contested decision out of day-to-day operations. The post says Ai2 is still iterating on the budget review process, with frequent chances for researchers to make their case and decisions made by managers who have the most context.