Goodfire Shapes AI Training Through Internal Representations

Right answers for the wrong reasons

Goodfire starts from a familiar weakness in AI training: models are rewarded for correct outputs, but good scores do not show that a model reached them for the right reasons. The company says this is especially relevant given recent reward hacking incidents, though it appears in simpler forms too.

Its illustrative example is a model trained to diagnose pneumothorax, a collapsed lung. In the training images, those labeled as pneumothorax often show a tube, which is how the condition is treated after diagnosis. A model can learn to rely on the tube instead of the lungs, and would then fail on patients who have not yet been treated. Goodfire names two comparable problems in language models: sycophancy, where models agree with users rather than answering accurately, and reward hacking, where they exploit graders instead of solving the problem.

How mechanistic shaping works

The framework adds a penalty to the training objective based on a concept the model represents internally. Goodfire describes two methods:

• Reliance shaping penalizes a model's dependence on a concept: "don't rely on this."
• Expression shaping penalizes the model for expressing a concept: "don't do this."

The steps are to find the undesirable concept with interpretability tools, add a loss penalty that reduces reliance on or expression of it, and train the model. Optionally, the concept direction is recomputed every n steps in case the representation shifts during training. Finally, the effect on the concept's monitorability is assessed. According to Goodfire, beyond standard training requirements the method needs only a concept direction and a layer that can be read.

In a small demonstration, the team trained a dog classifier on top of a frozen vision model and set rules written in the model's own concepts, such as classifying using only the nose, or using the tongue if available and otherwise the ears.

Results on chest X-rays and language models

For the medical test, Goodfire used a dataset of real chest X-rays and made the shortcut as tempting as possible. It kept only images of collapsed lungs with a chest drain and healthy lungs without one. The resulting model looked accurate in training but failed on collapsed lungs without a drain. The post says an LLM agent labeled sparse autoencoder features and scored them for confounds, and found tube-like features at many positions and orientations. According to Goodfire, the resulting automated pipeline can exceed the accuracy of the group-balanced data soft-ceiling for the setup.

In language modeling, the company says it prevented models from becoming sycophantic during post-training, and separately prevented reward hacking, while maintaining capability improvements in both cases. The paper also covers additional vision experiments and language model harmfulness reduction.

The monitoring question

A central concern is that training against interpretability signals can degrade the monitors themselves. The model's representations may come to circumvent the monitor while the targeted behavior persists. Goodfire studies this theoretically and empirically, describing how internals can change under shaping: regrowing, re-encoding, relocating and erasing. It also introduces a five-level hierarchy for measuring if and how a concept can be monitored after training.

The authors call their results encouraging: across their experiments the concept remained monitorable after shaping, though it often required a retrained probe. They add that they have not covered every setting and outcome.

What it means

For developers, the work points to a way to steer training when changing the data is too expensive or does not work, by targeting the concepts a model relies on. Goodfire says it is doing this work because better ways to align models are urgently needed, and describes the results as a promising set of tools and analyses. The caveat is monitoring: teams using such methods may need to retrain probes to keep tracking the concepts they shaped.