Astera Labs: Single Scorpio X Switch Lifts 8-GPU Inference Up to 89%

Why the fabric switch matters for inference

Astera Labs argues that inference now dominates AI compute. Citing Deloitte, the post says inference accounts for roughly two-thirds of all AI compute in 2026, up from about a third in 2023. As a result, hyperscalers are increasingly focused on generating profitable tokens per second.

According to the company, the switch that links multiple GPUs into one unit of compute is a significant conventional bottleneck for token throughput. If the switch has enough radix and bandwidth to give every GPU single-hop, line-rate access to every other GPU, compute can drive throughput with minimal overhead. If not, the fabric itself becomes the limit.

Astera Labs says conventional PCIe fabric switches max out at around 144 lanes. An 8-GPU cluster on that design needs two switches, each connected to its own CPU host, with the GPUs split into two four-GPU domains. Traffic between the domains has to detour through a CPU root complex, which adds a host-bridge hop and possibly a cross-NUMA node inter-socket transfer. According to the company, this limits bisectional bandwidth and adds latency to every transaction that crosses the boundary.

How the test was run

The company built two 8-GPU platforms that it describes as equivalent everywhere except the fabric. One used a single Scorpio X-Series 320L switch. The other used the conventional dual-144L topology. Astera Labs measured output tokens per second across a concurrency sweep from 1 to 512 requests, for three workload profiles and four large language models.

• Decode-heavy: 128 tokens in, 1,024 tokens out (1:8). Typical of conversational assistants and long-form drafting, and bound by per-token latency.
• Prefill-heavy: 3,000 tokens in, 3 tokens out (1000:1). Typical of content screening, classification and extraction, and both compute-bound and bandwidth-bound.
• Balanced: 3,500 tokens in, 500 tokens out (7:1). Typical of agentic workflows and retrieval-augmented answers, where prefill and decode contend with each other.

What Astera Labs reports

According to Astera Labs, the Scorpio X-Series 320L platform delivered more output tokens per second than the dual-144L baseline for every model, workload type and concurrency level tested. Gains reached up to 89%, depending on model and workload.

At the highest concurrency tested, the company says its advantage held at 30% or better for every model-workload combination. Dense models, which push more of their I/O through all-gather and reduce-scatter collectives, saw the largest gains.

The advantage was most consistent for prefill-heavy workloads, where Astera Labs says it held at 38% or better across the full concurrency sweep for every model. For decode-heavy traffic, the company says single-hop connectivity translated directly into more completed tokens per second. For balanced workloads, it says higher cross-sectional bandwidth kept large-context reads from stalling other users' decode. These figures come from the company's own testing.

What it means for AI factory economics

Astera Labs concludes that wherever a workload forces GPUs to move data across the fabric, a higher-radix, single-hop switch produces more tokens per second on the same GPUs at the same power envelope. For hyperscalers and neoclouds weighing every watt and port against the cost of a token, the company says this bears directly on the profitability of their AI factories.

The post adds that the test is one of several results from work on switches, interconnect and memory. It points readers to a separate blog on using Leo X-Series Smart Memory Controllers for KV cache offload to improve time-to-first-token and tokens per second.