Redwood Research: Capabilities work can shift the safety-usefulness frontier too

Why the definition breaks down

The post starts from a tempting definition: safety research lets developers deploy an AI system more safely without making deployment much more expensive or much less useful. Visually, this means pushing out the safety-usefulness Pareto frontier, so that at any level of usefulness, more safety is available.

The author points out the awkward consequence. Performance optimization for inference, for example, makes it possible to use weaker, safer models more extensively, which pushes the frontier outward. More generally, the post says, "any successful research whatsoever pushes out this Pareto frontier because research can only ever create more options." If a definition counts seemingly all capabilities research as safety research, the author argues, something has gone wrong with it.

The core observation is that developers must choose a particular point on the frontier, and some technological improvements tempt them to sacrifice safety.

Two different shapes of progress

According to the post, safety research typically creates new options for safety without creating substantial new options for usefulness, pushing the frontier to the right. Capabilities research typically creates new options for usefulness at the expense of safety, pushing the frontier up and to the left.

The post defines safety as the probability that the model does not take over, and notes that a safety of 1 can always be reached by not producing or deploying models at all. It adds that safety research is not purely about safety either. Fixing reward-hacking often improves usefulness, while most methods of improving monitorability degrade it. Developers also need not be altruistic to care about safety, since they do not want to be taken over by AIs either.

Because safety improvements make the usefulness cost of safety relatively smaller, the author expects them to lead rational developers to choose higher safety than capabilities improvements do. The post says this can happen in both directions in principle, and gives the example that developing neuralese monitors might tempt rational developers to use neuralese.

RLVR environments versus inference speedups

The post gives two examples of capabilities research. The first is building many RLVR environments, many of them hackable. This makes it easier to train a more useful model but barely easier to train a safer one. Developers could use the larger pool to filter out hackable environments more aggressively and get a somewhat safer model, but the author says current frontier developers would instead train a more capable and dangerous model using most of the new environments.

The second example is speeding up inference throughput, which the post credits to Buck. At a given level of usefulness, the author says it is probably safer to use a weaker model with more inference-time compute. The author would rather do AI research with 100 parallel instances of a model that can probably be controlled and whose alignment is more trustworthy, than with one instance of an equivalently useful model lacking those properties.

Inference speedups can also strengthen control protocols. For a protocol that resamples suspicious actions, safety depends on how many actions are resampled per trajectory, and higher throughput allows more resampling at fixed usefulness. The post also suggests pretraining and other predictive training tasks may be substantially less dangerous than outcome-based RL.

What it means

The author stresses the post is not an argument for more people to do capabilities research. In fact, the post says capabilities research at any frontier AI company is currently probably bad. It treats the question as a puzzle to sharpen models of when safety and capabilities research are good.

The post does describe a plausible future regime with much higher political will than today, in which certain kinds of capabilities research could be an effective way to improve safety. It also sets aside a separate category of interventions that make developers more willing to give up usefulness for safety, which it calls crucially important, especially now.

For researchers and funders, the takeaway is that judging work by whether it expands the frontier is not enough. The shape of the expansion, and the incentives it creates for developers, matter too.