IBM's Spyre Accelerator Moves to PyTorch Inductor With a Compiler 7x Smaller

From a custom stack to Inductor

Until recently, torch-spyre relied on sendnn, a separate compilation stack built entirely from scratch. It now uses an open-source backend that extends Inductor, the compiler inside PyTorch's torch.compile. The authors say most of Inductor carried over untouched, and that the real work concentrated in three places where Spyre's hardware asks a fundamentally different question of the compiler than a GPU does.

They also stress that this was not a drop-in swap. Substantial out-of-tree extensions were still needed at Inductor's lower levels and in code generation.

IBM designed Spyre as a clean-sheet AI accelerator for enterprise inference workloads, prioritizing power efficiency. According to the post, it is a 75-watt, PCIe-attached accelerator that requires no custom power delivery or cooling. It is a dataflow accelerator built from 32 small cores wired together over a ring, each with its own local 2 MB scratchpad rather than a shared cache. Data moves in fixed-size chunks called sticks, which are 128 bytes, or 64 values in fp16.

How the compiler is structured

The compiler has two stages. The first uses Inductor extensions, which live in the torch-spyre repository, to lower a torch.compile'd graph to an intermediate representation called SuperDSC. The second uses a proprietary compiler called DeepTools to translate SuperDSC into instructions that run on Spyre hardware. The post focuses on the first stage.

The extensions plug into the standard pipeline, in which Dynamo traces the model, AOTAutograd partitions it, and Inductor lowers the FX graph into its LoopLevelIR. Torch-spyre adds three extension points:

• Decompositions: Spyre-specific op decompositions at the FX graph stage, for example mapping layernorm to sequences of primitive Spyre operations.
• Lowerings: Spyre-specific op lowerings from the FX graph to LoopLevelIR.
• CustomPreSchedulingPasses: the bulk of the Spyre-specific compiler, covering layout assignment, tiling, work division and scratchpad allocation.

The last of these is currently wired in through a monkey-patch on Inductor's GraphLowering.codegen(), not a first-class hook. Inductor's own _pre_fusion_custom_pass runs too late, the authors say, because it only runs after LoopLevelIR has been turned into SchedulerNodes, and Inductor does not support rewriting the graph at scale after that point. The team is working with upstream PyTorch to formalize these extension points so other out-of-tree hardware backends can use them.

Three hardware differences behind the design

The post says Inductor's mental model of hardware is shaped by GPUs, and three pieces of it do not carry over to Spyre.

Sticks and layouts. Spyre cannot address memory at the level of individual elements. The stick, a 128-byte aligned chunk, is the unit of data movement and computation, so Inductor's index arithmetic must reason in sticks. A pointwise op requires all its inputs and its output to share one stick dimension, the innermost one. When inputs arrive with incompatible sticks, the compiler must insert a restickify, a data-movement op that reforms the sticks themselves. Choosing stick dimensions across a whole graph to minimize data movement is a global optimization problem. Layout also affects speed: on a batched matmul with all tensors already in device memory, changing how input and output tensors are tiled cut kernel time by 29-69% across a range of batch sizes and shapes, according to the authors.

Scratchpad memory. Each core has a small local scratchpad, and behind it sits only global device memory. Every transfer happens because the compiler emitted an explicit instruction. With no hardware-managed layer, a value not planned to stay in scratchpad must be written back and fetched again. Since workloads on Spyre are frequently memory-bound rather than compute-bound, the post says that round trip is often where the time goes.

Core assignment. Spyre's cores run as a static dataflow machine, with no GPU-style runtime scheduler balancing load across streaming multiprocessors. The division of work across cores therefore has to be decided at compile time.

What it means

For teams building accelerators outside the PyTorch tree, the post offers a case study: a new chip with a very different execution model could reuse most of Inductor and concentrate effort on a few hardware-specific passes. The authors say Inductor was "exactly the right substrate" for the work, and the roughly 7x smaller compiler is their measure of the leverage gained. The planned upstream work on formal extension points, if completed, is meant to make that approach easier for other hardware backends.