AWS shows two-node LLM serving with Ray LLM Deep Learning Containers

Why split prefill and decode

According to AWS, LLM inference has two distinct phases. Prefill processes the whole input prompt in parallel and builds the key-value (KV) cache. Decode then generates the response one token at a time, reading and extending that cache. Prefill is compute bound, while decode is bound by memory bandwidth.

When both phases share one GPU, they contend for the same resources, and bursty prefill work interferes with latency-sensitive decode work. Prefill/decode (PD) disaggregation places the phases on separate nodes so each runs without interference and can be scaled independently. AWS says this keeps compute-heavy long prompts from stalling token generation for requests already in progress, and names Retrieval Augmented Generation (RAG), document summarization and agentic applications as suitable workloads.

The trade-off, AWS notes, is that the KV cache built during prefill must be handed to the decode node, and the transfer has to be fast enough not to erase the benefit. That is why the interconnect between nodes matters.

What the Ray LLM DLC contains

AWS Deep Learning Containers are pre-built, performance-optimized Docker images that bundle a framework, its dependencies and the GPU stack into a tested, patched combination. According to AWS, all components are validated together before each release, so the CUDA runtime, framework and serving layer do not drift apart, and security patches are applied at build time.

The Ray LLM DLC applies this to distributed serving. It ships:

• vLLM as the inference engine
• Ray Serve HTTP as the serving layer
• NIXL (NVIDIA Inference Transfer Library) for KV-cache transfer between nodes
• The EFA stack (libfabric and aws-ofi-nccl) for high-bandwidth node-to-node communication

AWS says customers otherwise have to assemble this dependency chain themselves and debug it when components fall out of alignment, work it calls undifferentiated. The post follows an earlier one that introduced the Ray Serve DLC on a single GPU node.

How the deployment is laid out

The example runs on an EKS cluster inside a virtual private cloud. A CPU system node (m7i.xlarge) hosts system workloads, the KubeRay operator and the CPU-only Ray head pod. Two GPU worker nodes (g6.8xlarge) run the Ray workers, one for prefill and one for decode. Both pull the DLC from Amazon Elastic Container Registry.

KubeRay watches the RayService that the user applies and wires the head-to-worker join automatically. Ray Serve places the prefill and decode deployments on the two GPU workers, and the KV cache moves from prefill to decode over NIXL, using EFA through NIXL's libfabric backend. The GPU node group is pinned to a single Availability Zone because EFA cannot cross AZs.

Prerequisites include an AWS account, enough service quota for two g6.8xlarge instances (at least 64 vCPU), and the AWS CLI, kubectl, eksctl, helm and envsubst tools.

Configuration with no application code

Because Ray Serve LLM has a built-in application builder, the whole serving application is a block of YAML, according to AWS. There is no handler class, model archiver or custom image. A helper called build_pd_openai_app takes a prefill configuration and a decode configuration and connects the KV-cache transfer behind one OpenAI-compatible endpoint.

The example serves Qwen/Qwen3.5-9B with max_model_len set to 4096, bfloat16 precision and gpu_memory_utilization of 0.90. Both phases use NixlConnector with kv_role set to kv_both and distinct engine IDs. AWS stresses one setting: kv_connector_extra_config with the LIBFABRIC backend. Without it, NIXL defaults to UCX, which AWS says runs at TCP speed even on EFA.

Each worker pod requests one GPU and one EFA interface. The complete code is available in the accompanying repository.

What it means for users

For teams running large or high-throughput models on AWS, the container removes much of the manual work of matching GPU libraries, networking and serving software across nodes. AWS says the same approach also suits tensor, pipeline and data parallelism, though this walkthrough covers only the two-node PD setup. The example uses a small 9B model on modest GPU instances, so it serves as a template rather than a sizing guide.