Red Hat AI 3.5 lets chat and batch share one GPU without latency hits

Why separate GPU pools cost so much

Red Hat frames the problem as a question every shared inference platform eventually faces: whose request goes first? One foundation model on a cluster may serve a customer chatbot, a batch summarization pipeline, an agentic retrieval-augmented generation (RAG) workflow and an internal copilot. They share a model but not performance requirements.

A chatbot needs its first token in under a second. A batch pipeline cares about throughput per dollar. A pool with no notion of priority uses one queue and one dispatch order, so a batch flood during a traffic spike makes chatbot requests wait behind it.

The usual answer is a separate GPU pool per workload type, each sized for its own peak. Red Hat says this means every pool carries its own reserve, and no pool can borrow another's idle capacity.

How priority-aware serving works

The feature is built on llm-d flow control, a layer in front of the model servers. Applications keep calling the same endpoint. The operator assigns each workload a priority in an InferenceObjective Kubernetes resource, sets a saturation threshold (the point at which the pool counts as full) and picks a fairness policy. The gateway reads a caller's credentials and tags each request with a priority and tenant.

Once the pool saturates, three rules decide who goes next:

• Priority: the highest band with waiting requests goes first.
• Fairness: within a band, dispatch rotates across tenants so one tenant's burst does not crowd out its peers.
• Ordering: within a tenant's queue, requests are served first-come, first-served.

Below the threshold, spare capacity is never held back.

Test results from Red Hat

On Red Hat AI 3.5, four tiers shared an NVIDIA H100 serving gpt-oss-20b under the same traffic. Only the detector changed, meaning the part of flow control that decides when the pool counts as full. Time to first token (TTFT) with the new concurrency detector was 364 ms for Platinum (realtime), 430 ms for Gold (realtime), 699 ms for Silver (standard) and 3.0 s for Bronze (batch). The older utilization detector gave 1.3 s, 1.5 s, 2.0 s and 2.7 s respectively.

Without flow control, a batch flood on the same GPU drew 48,224 rejections across 3 runs, each a 429 too many requests error. With flow control there were none, and requests waited their turn at every output length tested.

Within a tier, three premium tenants shared a band while one burst to 82 requests per second and the other two held at 6. The quiet tenants' wait stayed flat at about 17 milliseconds, while the burster waited 45 milliseconds and was still served.

In internal testing with premium, standard and batch on four NVIDIA A100s, premium's first token stayed between 2.1 and 2.8 seconds in every scenario. Red Hat says that compares with 6.7 to 8.3 seconds with the earlier detector.

Limits and tuning

Red Hat says there is one number to tune. A threshold set too high lets requests pile up inside the model servers. One set too low starts queueing while GPUs still have room. Its suggested method is to raise load until premium misses its latency target, then set the threshold to a fraction of that value.

The company lists two limits. Priority is strict, so sustained premium load holds batch work back for as long as it lasts. And when responses run to thousands of tokens, requests already past the gate dominate latency, so flow control can no longer guarantee premium protection. Red Hat advises testing long-output agentic or reasoning workloads first. Teams whose traffic is all one kind, or whose pool never fills, do not need the feature.

What it means for platform teams

Red Hat gives an illustration: three pools of 8 GPUs at 30% average utilization amount to 24 GPUs doing roughly 7 GPUs' worth of work. Putting batch at low priority in a shared pool lets it use whatever interactive tiers leave free. How much capacity this frees depends on how peaks overlap, and Red Hat suggests checking against your own utilization data.

The company says the next step is SLO-aware serving, where an operator states a latency target per tier instead of translating it manually into a saturation threshold. According to Red Hat, that is where llm-d goes next.