Goodfire Releases First Sparse Autoencoders for DeepSeek's 671B R1 Model
Goodfire has trained what it says are the first public sparse autoencoders (SAEs) on a true reasoning model, DeepSeek's 671B parameter R1, and has open-sourced them. The interpretability company also reports two unusual findings about how R1 responds to steering.
- Goodfire trained two sparse autoencoders on the 671B parameter DeepSeek R1 and open-sourced them.
- One SAE used a custom reasoning dataset, the other OpenR1-Math; both datasets and SQL databases of max activating examples are available.
- Steering R1 at the start of its chain of thought fails; it works after a prefix like "Okay, so the user has asked a question about…".
- Oversteering some features makes R1 revert to its original behavior before outputs turn incoherent.
- Goodfire says the findings are early and that SAEs have known limitations.
Two SAEs for a very large model
According to Goodfire, the release includes two SAEs for R1. The first was trained on R1's activations on a custom reasoning dataset, which the company is also open-sourcing. The second used OpenR1-Math, a large dataset for mathematical reasoning. Goodfire says these datasets let it find the features R1 uses to answer challenging problems.
The company describes the SAEs as the first public interpreter models trained on a true reasoning model, and on any model of this scale. It says the undistilled 671B model is an engineering challenge to run, and that training relied on its custom inference engine and interpreter model training infrastructure, which it says enables interpretability at the half a trillion parameter scale.
Because R1 is hard for most independent researchers to run, Goodfire also uploaded SQL databases containing the max activating examples for each feature. Usage instructions are on the project's GitHub. The team says its SAEs learned many features that reconstruct behaviors core to reasoning models, such as backtracking, and it published curated examples from the general reasoning SAE along with an interactive UMAP feature map built with DataMapPlot.
Two surprises when steering R1
Much of Goodfire's early work focused on steering R1 by changing the strength of particular features. The company says it has not yet systematically investigated why, but it highlights two behaviors it has not seen in non-reasoning models.
Steering has to start later. Models are usually steered from the first token of the response. Goodfire reports that doing this at the beginning of R1's chain of thought fails. Effective steering only worked after the model began with something like "Okay, so the user has asked a question about…". The team also saw attention sinks, tokens with far higher than normal average activation, toward the end of this prefix rather than at the very start of the response, as is normal. Goodfire's hypothesis is that such phrases were so common in R1's training reasoning traces that the model treats them as part of the prompt. It notes that more than 95% of R1's English reasoning traces begin with "Okay,".
Oversteering reverts behavior. In a typical model, pushing a feature ever higher makes outputs fixate on that concept until they become incoherent. With some features in R1, Goodfire observed that oversteering paradoxically makes the model return to its original behavior. Outputs do become incoherent if the feature is pushed further, but only after the steered behavior disappears. Examples in the post include swapping an operator in a math problem and reducing the time the model spends thinking.
Possible explanation and caveats
Goodfire's working hypothesis is that R1 implicitly recognizes a state of confusion or incoherence when its activations are overly perturbed, and then stops and course-corrects. It suggests that training of reasoning models might incentivize greater implicit awareness of their internal state, pointing to the fact that they often backtrack when a line of reasoning is not working.
If this is a general property of reasoning models, the company says, attempts to change their behavior, for example to suppress dishonest responses, may require more sophisticated techniques, since models may find ways to route around modifications. Goodfire stresses that more work is needed and that the findings are early.
The company also acknowledges that it does not believe SAEs solve the overarching problem of mechanistic interpretability, citing a number of known limitations, though it calls them a central part of today's toolbox.
Why it matters
Goodfire argues that as reasoning models become a foundation for advanced AI systems and possibly act more agentically, understanding their internal mechanisms is crucial. It lists benefits such as better understanding of capabilities and limitations, monitoring and debugging of unexpected behaviors, more targeted safety interventions, and greater transparency and trust for users.
For researchers, the practical value lies in access: open SAEs, a reasoning dataset and precomputed activation examples let teams study R1 without running the full model. Goodfire says it plans more work on reasoning models and is keen to see what others find with its tools.
Goodfire Research: Under the Hood of a Reasoning Model - Goodfire
