PyTorch serves Meta’s 816B Muse Spark Open on one TPU v7 host

One program replaces many kernel boundaries

Muse Spark Open has 816B parameters in total, with about 42B active per token. All 62 decoder layers are mixture-of-experts layers. Each layer routes every token to 8 of 256 experts, while the model combines sliding-window attention over the last 2,048 tokens with full-context attention reaching up to 1M tokens.

PyTorch says its earlier serving path ran each layer as several kernels and collectives. Those program boundaries interrupted the flow of weights from high-bandwidth memory. The new megakernel runs the entire decode or verify step as one Pallas program on each device, while SGLang continues to handle surrounding work such as the embedding, draft model, language-model head and sampling.

Inside the megakernel, weight loads, KV-cache reads and cross-device transfers are pipelined with computation. Dense weights are streamed through a ring of tiles, selected expert weights are requested as soon as routing finishes, and remote DMA operations move partial results between devices without waiting for separate collective operations.

Reported decode and verification gains

According to PyTorch, the megakernel halves the time per output token for one or two concurrent requests, where the optimization helps most. The advantage becomes smaller as concurrency increases. In a measurement using DFlash2 speculative decoding, the megakernel was 1.94 times faster per output token than the native path for one request, at 1.95 ms compared with 3.78 ms, and 1.85 times faster for two concurrent requests.

PyTorch says the native verify step took 2.9 times as long as streaming the step’s distinct experts and dense weights from HBM. The optimized verify step runs within 1.5 to 1.7 times that weight-streaming time. The post attributes the improvement to reducing overhead from collectives and keeping memory transfers in flight while the matrix units compute.

The TPU v7 setup uses eight exposed devices, because each of the host’s four chips has two chiplets. The model is sharded eight ways with tensor parallelism, with one shard assigned to each device. Each chip has 192 GiB of HBM, and the four-chip host has 960 GB of DRAM for the L2 HiCache tier.

Agentic coding benchmark and serving tradeoffs

PyTorch also reports results from the 1M-context version of SemiAnalysis’s AgentX agentic-coding benchmark. The benchmark is built from 393 opt-in Claude Code sessions and replays request lengths, prefix sharing between turns, tool pauses and subagent structure, using synthetic tokens instead of the original content.

On the reported curve, the TPU v7 system delivered 433 output tokens per second per request at 6,101 tokens per second per chip, rising to 25,906 tokens per second per chip at 95 output tokens per second per request. Every TPU point in the figure completed requests within 47 seconds at p90. The comparison uses InferenceX external owning costs of $1.21 per TPU v7 chip-hour and $1.86 per GB200 GPU-hour, with a GB200 baseline based on InferenceX’s published results as of 2026-11-02.

For users and serving operators, the work shows how the TPU v7 implementation targets both low-concurrency decode latency and longer-context workloads. PyTorch also describes additional optimizations for KV-cache capacity and prefill, including sequence-on-lane KV layout, L2 HiCache with session-aware eviction, a fused MoE kernel and sequence parallelism.