Goodfire argues individual neurons can explain how LLMs compute
Goodfire Research says individual neurons in a large language model can serve as meaningful units for understanding computation, challenging the view that researchers must first transform models into learned feature spaces. Its study of Qwen3.6-35B-A3B combines sparsification, attribution tools and research agents to examine how neurons function.
- Goodfire studied neurons in Qwen3.6-35B-A3B.
- Ablating 92.2% of the weakest MoE activations increased loss by 2.9%.
- Neuron descriptions predicted firing with 95% accuracy versus random text.
- Descriptions recovered 73% of ablation damage against a restorable-damage ceiling.
- Goodfire says neurons can be useful units for interpreting LLMs.
A case for studying neurons directly
Neurons are hidden units in the multilayer perceptron layers of a transformer. They were among the first units used to interpret neural networks, but research on polysemanticity, where one neuron appears to respond to seemingly unrelated inputs, made them look difficult to study directly.
That concern helped drive work on learned feature spaces and overcomplete feature dictionaries. Goodfire argues that this approach adds a step before scientific analysis can begin, because the model must first be transformed into a different representation. The company’s research instead examines the circuits in which individual neurons participate, including the neurons that drive their activation and those that use their output.
Three properties make neurons tractable
Goodfire reports three findings. First, the neurons in its study are already functionally sparse. Ablating the 92.2% weakest mixture-of-experts neuron activations for each forward pass produced what the company describes as modest effects on model capabilities, including a 2.9% increase in language model loss.
Second, most neurons have few computational roles. The majority have either one mode or a single mode that is overwhelmingly causally significant. When neurons have multiple causal modes, Goodfire says the number is small and its methods can describe them clearly.
Third, research agents can produce natural-language descriptions of neurons. According to Goodfire, giving those descriptions to a judge model enabled it to predict whether a neuron would fire in a passage with 95% accuracy versus random text. The descriptions also simulated token-by-token behavior in high-activation regimes well enough to recover 73% of the damage from ablation, measured against a ceiling of restorable damage.
What the findings mean for interpretability
The research does not claim that every neuron represents one concept. Goodfire says some neurons are genuinely polysemantic and that some behavior remains unexplained. Its argument is narrower: important firings tend to have a small set of computational functions, and those functions can be captured by concise descriptions.
The company also reports that neurons exert most of their downstream influence at strong activation levels, above the 97th percentile at the positive extreme and below the 3rd percentile at the negative extreme. These high-activation events are both more important to the model and easier to describe, according to the study.
If the result generalizes beyond the model examined, researchers could use individual neurons as a direct basis for interpreting language models rather than relying exclusively on learned proxies. For users, the immediate significance is indirect, but better neuron-level explanations could support more precise investigation of how models produce their outputs and how particular behaviors arise.
Goodfire Research: Neurons as an Interpretable Basis - Goodfire
