AMD MI355X Beats NVIDIA B300 per GPU on Kimi K3 AgentX Benchmark
An eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM delivers 24.4% more total tokens per second per GPU than NVIDIA HGX B300 at 50 tokens/s/user P90 interactivity on the AgentX benchmark, according to the vLLM project. Under the benchmark's ownership-cost model, it delivers 87.4% more tokens per dollar.
- Eight AMD Instinct MI355X GPUs running Kimi K3 on vLLM deliver 24.4% more total tokens/s/GPU than NVIDIA HGX B300 at 50 tokens/s/user P90 interactivity.
- About 7,567 versus 6,083 total tokens/s/GPU, per the SemiAnalysis InferenceX October 9, 2026 snapshot.
- Using SemiAnalysis's modeled TCO of $1.50 versus $2.26 per GPU-hour, AMD yields 87.4% more tokens per dollar.
- Optimizations include DCP8 for MLA history, CPU offload of KV and KDA state, and removing duplicate MoE projection work.
- A memory-saving change (#59591) raised throughput 28.9% at concurrency 70 in a separate test.
The AgentX results
AgentX is a public benchmark that replays real coding-agent session traces with synthetic tokens. It keeps request lengths, timing and prefix-sharing patterns, and offers over 96% theoretical prefix reuse with unlimited cache, according to the vLLM post.
At 50 tokens/s/user P90 interactivity, the AMD Instinct MI355X serves approximately 7,567 total tokens/s/GPU, compared with 6,083 for NVIDIA HGX B300. That is the 24.4% throughput advantage. Using SemiAnalysis's modeled TCO of $1.50 versus $2.26 per GPU-hour, the figures translate to 18.16 million versus 9.69 million total tokens per dollar of TCO.
The numbers come from the SemiAnalysis InferenceX October 9, 2026 snapshot. Interactivity is the reciprocal of P90 response inter-token latency and excludes time to first token. Total tokens include reused input and generated output. The post notes that the public curves compare complete serving stacks, and that the measurements beside each optimization come from separate tests.
On paper, the two chips are close. Published dense matrix peaks are 10.1 PFLOP/s MXFP4 for the MI355X and 13.5 PFLOP/s FP4 for the B300. Both have 288 GB HBM3E and 8 TB/s peak memory bandwidth. The post credits sustained software engineering by AMD and collaborators across the vLLM community.
Why Kimi K3 is hard to serve
The post describes a model that mixes two kinds of attention. Its 24 MLA layers retain a compressed KV history that grows with the conversation. Its 69 KDA layers carry forward state matrices with fixed dimensions and update them as new tokens arrive. In MoE layers, a router selects expert networks to transform each token.
A returning agent turn can reuse most of its prefix, but processing a short continuation still touches all three kinds of data. The optimizations are grouped by where the waste occurs: across turns, within a model step, and in the choice of draft depth for speculative decoding.
Cache and MoE changes
The first change targets cache memory. Under TP8, each GPU handles 12 of Kimi K3's 96 MLA query heads but stores the entire compressed KV history. With eight-way decode context parallelism (DCP8), each GPU stores one eighth of the cached positions. During decode, the GPUs exchange queries, compute against their local history and merge partial answers. For a 142,000-token conversation, the post estimates the MLA payload at about 1.96 GB per GPU, which DCP8 reduces to about 245 MB, before allocation overhead.
When state no longer fits in GPU memory, native vLLM CPU offload keeps MLA history and KDA checkpoints in host DRAM. A cache hit needs matching resume points. If MLA history reaches token 3,072 but the latest compatible KDA checkpoint is at 2,048, the request must resume at 2,048 and recompute the remaining 1,024 positions. Saving the attention-block tail alongside the KDA checkpoint aligns the resume positions (#54736).
The second change removes duplicate work. In the original ROCm path, the routed-expert output projection ran eight times, once on each GPU. Output coordinates are now divided among GPUs and added to the shared-expert partials before the existing all-reduce, with no new collective (#51253). In the PR's test on eight MI355X GPUs, output throughput rose 3.05, 4.91% across concurrency 8, 128.
A follow-up (#59591) loads only the rows each rank uses, freeing about 3.85 GiB per GPU across 92 MoE layers at TP8. That landed after the image used for the headline result. In a separate AgentX experiment it showed less than 1% higher throughput at concurrency 32 and 48, but 28.9% at 70, where the reported prefix-cache hit rate rose from 74.8% to 80.1%. The post calls the larger gain at high concurrency consistent with cache pressure and notes that repeated-run uncertainty was not reported.
What it means
The result is a vendor-adjacent claim built on a public benchmark, and the post stresses that it compares complete serving stacks rather than isolated hardware. Within that scope, it shows that software work on caching, parallelism and kernels can close the gap between two accelerators with similar memory specifications. For teams running long-context coding agents, the post's main lesson is that prefix reuse and cache capacity weigh heavily on throughput, especially as load rises.
