Redwood questions whether Gemini 4 training suppresses chain-of-thought signals

A disputed description of chain-of-thought training

In a post published on Nov 10, Alek Westover and Buck Shlegeris said the Gemini 4 Safety report appears to state that some reinforcement learning training uses model thoughts as context. The report says that “Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs,” according to the quotation reproduced by Redwood Research.

The report also describes these auxiliary rewards as low-risk because they are “either orthogonal to or structurally aligned with the output reward.” Redwood interpreted this as meaning that Google DeepMind, referred to in the post as GDM, was training on chain of thought with auxiliary rewards, possibly by having a language model assess partial reasoning and reward reasoning judged to be on track.

Redwood said that approach concerns it because judges could be biased against reasoning that signals misalignment. Statements such as “I will now try to evade the monitor” or “I will solve the problem via hacking” could be penalized even without explicit instructions to reject such thoughts. The researchers argue that this could suppress verbalized misalignment and reduce chain-of-thought faithfulness.

Conflicting claims leave researchers seeking details

Redwood said GDM has also written that “we always follow one rule: we don’t penalize the model for what it writes in its chain of thought during training”. The researchers added that conversations with GDM staff indicated they were not training on chain of thought, although the staff members could not explain the technical details.

This leaves Redwood unsure whether the safety report describes the same practice that GDM has denied. The post also considers whether GDM might avoid penalizing bad reasoning while increasing the weight of good reasoning. Redwood argues that, in many reinforcement learning algorithms, including GRPO, those operations can be equivalent, although the amount of training pressure could differ depending on which reasoning receives rewards.

Redwood is also skeptical of the report’s use of “orthogonal” rewards. The researchers say a reward can meet the cited definition of orthogonality while still encouraging a model to avoid mentioning misaligned actions. In their view, avoiding direct training against a monitor does not by itself establish that chain-of-thought integrity has been preserved, because a reward model that sees the reasoning could also teach a model to hide it.

Why monitorability matters to the safety argument

Redwood said GDM’s monitorability evidence does not resolve its concerns. The researchers point to Argon completing LinuxArena side tasks without being caught by GDM’s production monitor about half the time, while also having high no-chain-of-thought capabilities. They said the GDM safety report and Alek disagree about how monitorable Argon is, and that they plan to discuss the issue in a later post.

The researchers also argue that monitorability evaluations may miss broader benefits of chain of thought, including helping people understand the reasons behind model behavior. A monitor might detect risky behavior from a model’s actions alone, they said, without showing whether training has made the reasoning less informative.

Redwood’s best guess is that the apparent contradiction reflects a misunderstanding, and that GDM is not intentionally training on chain of thought in a particularly concerning way. But the researchers said they would like clarification, arguing that greater transparency from AI companies would make it easier for external researchers to assess the safety effects of their training practices.