Ai2's StateBench Shows Frontier Models Struggle to Track State Over Long Inputs

What StateBench measures

State tracking covers tasks such as reading code, where variables are assigned and reassigned, or following a story in which characters move and objects change hands. Ai2 gives the example of a mystery novel in which "John gave Mary the knife" right before a murder, so the state of each character has changed. Harder versions require following a chain of changes, where one missed step can throw off the final answer.

Ai2 notes that transformers have well-documented theoretical limits on hard state tracking with long dependency chains. That has helped motivate recurrent and hybrid architectures such as Gated DeltaNet, which carry a running memory forward. Existing tests tend to focus on the hardest cases, so Ai2 built StateBench to cover problems of different complexity and to show which kinds of state tracking each architecture can handle.

Four levels of difficulty

The benchmark has four levels, which Ai2 explains through types of variable assignment:

• R-trivial: immutable assignment. A value never changes, so the model only needs to recover what it was first set to.
• Aperiodic: reassignment by value. The model finds the most recent value, like a thermostat setting.
• Soluble: self-referential updates such as x = x + 5, like a game score that builds a running total.
• General finite: assignment by reference, such as x = y or swapping variables. Ai2 compares this to a magician swapping cups to hide a ball, the S₅ problem of following swaps among five objects.

Ai2 says the levels reflect theoretically motivated distinctions rooted in Krohn, Rhodes theory, a branch of algebra. To compare levels fairly, it starts from a general finite sequence and derives three easier versions. All four use the same number of variables and distinct values, with assignments at the same points in the sequence. Only the type of assignment changes.

Results for pretrained and frontier models

Ai2 first tested open pretrained models, including two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller models with 275 million and 2.7 billion parameters. All were pretrained on the same data, and none had seen contexts longer than 64,000 tokens in training. Almost all performed well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems.

Problem format also mattered. The 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits. Ai2 gives this as a reason StateBench tests the same problem in several formats.

Ai2 also tested four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. They were asked to read each sequence of assignments and generate the final value of every variable. All four did well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply with length, dropping to 13% or lower on the longest general finite problems. Ai2 points to sensitivity as one likely factor: at the harder levels many more steps can affect the answer, leaving less room to miss an update.

Training directly on the task

To see whether the difficulty comes from architecture or from pretraining, Ai2 trained several 275-million-parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. They trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments, or 250 times the maximum training length.

Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble problems out to the longest sequences tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite problems. On the hardest level, the Gated DeltaNet variant with a more flexible memory update stayed near perfect out to roughly 10 times the longest sequences seen in training.

Ai2 concludes that for Gated DeltaNet and hybrid models the gap points to training rather than architecture, since trained directly they handled levels their pretrained counterparts could not. Transformers could never learn the hardest level even with direct training.

Open questions and what it means

Some questions remain open. Ai2 says its new proof shows that Gated DeltaNet with a more limited memory update (non-negative eigenvalues) can solve all aperiodic tasks and certain periodic tasks, such as tracking whether a repeated input has occurred an even or odd number of times, under the paper's fixed-precision arithmetic with rounding. Its exact place in the hierarchy is unresolved, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers affects it.

For researchers choosing between architectures, StateBench offers a way to see which kinds of state tracking a model can handle rather than only the hardest case. For users of current frontier models, the results suggest that tasks needing long chains of dependent updates remain a weak spot as inputs get longer. Ai2 links a tech report, data and code for the benchmark.