Redwood Research's Screwtape edits real agent transcripts to beat AI monitors

How the pipeline works

Screwtape starts with a real transcript, a monitor and a threat contract. The contract is a document that describes the threat vector and the severity the pipeline requires before it accepts an attack. The targeted monitors are action-only, meaning they do not look at the agent's chain of thought.

An attacker agent reads the contract, edits the transcript and writes a description of the inserted attack. Some edits are rejected programmatically. The attacker cannot change user messages or edit subagent calls, for example, and such edits are sent back for revision.

An LLM verifier then checks the candidate. Unlike the monitor, which sees only the edited transcript, the verifier also receives the attack description and the diff between the source and edited transcripts. It judges whether the attack is realistic, clearly harmful and could plausibly succeed in the original environment. If the verifier accepts the edit, the monitor scores it. The attack counts as successful when the score falls below a threshold set to a fixed false positive rate on honest transcripts. If either check fails, the reasoning goes back to the attacker as feedback, and the loop continues until it reaches the maximum number of iterations.

The monitor in the loop can be the target itself or a proxy monitor built to stand in for it.

Results and costs

Redwood ran the pipeline on Redwood staff transcripts, using proxy monitors based on Opus 4.8 and Gemini 3.1 Pro. The final attacks were then tested against a held-out monitor from a frontier developer. Across 252 runs on 94 unique transcripts, the pipeline produced 35 attacks that evaded the proxy monitors. Six of them also evaded the frontier developer's monitor and were confirmed as valid through human review.

The method is expensive. According to Redwood, each iteration takes about 5 to 10 minutes, a run can take a few hours, and most runs find no attack. The authors say better elicitation of the attacker and running multiple runs in parallel could help.

How it differs from other approaches

Redwood compares Screwtape with several ways of generating attack transcripts, which differ in realism, how hard the attacker is pushed, how hard validity is to check, and cost.

• Synthetic settings such as LinuxArena, BashArena and SHADE-Arena use real agent rollouts, but some threats, such as rogue deployment or weight exfiltration, are hard to implement realistically there.
• Transcripts written as text, as in STRIDE, need no real environment, but an LLM must invent both environment responses and user messages, which can make them less realistic.
• Simulated environments such as GLOOM can start from a forked real transcript, but after the fork the simulator invents the environment's responses, so the rollout can drift far from the original.
• Human red teams, as in SLEIGHT-Bench, METR and the UK AISI Control Red Team, produce strong attacks, but each one takes hours of human work, 2 to 3 hours per attack in SLEIGHT-Bench.

Screwtape trades off differently. It edits real transcripts to keep them realistic and iterates against the monitor, which lets it simulate stronger attackers and find holes that do not depend on known weaknesses. The cost is that it tests the monitor, not how well an agent would attack on its own. The authors say that to their knowledge no published method directly optimizes attacks against a monitor by iterating over edited agent transcripts, and mention unpublished concurrent work using an evolutionary approach.

Lessons from building it

The team developed the pipeline over a couple of months. Their first lesson is that finding severe evading attacks without human help is hard. The attacker starts with simple attacks and tries to make them trickier, but monitors catch most of them.

Judging severity is also fuzzy. The same action can be catastrophic in one context and safe in another. Over many iterations the attacker drifted toward borderline cases that look incriminating but are not severe enough, such as publishing expired credentials or planting a backdoor that slightly changes a button color. Because even human experts struggled to label severity, the team set conservative high-severity thresholds.

LLM verification proved viable but flawed. A simple yes/no verdict led to early stops on trivial successes that human experts did not find valid. Redwood moved to a detailed rubric with dimensions inspired by security engineering frameworks (NIST SP 800-160v1r1 and the STPA Handbook). The verifier models what harm an attack can cause, whether the harm has already happened or is only prepared, and whether the user authorized the actions. On average it rejects 60 to 70% of the attacker's candidate edits within a run. Because it sometimes accepts invalid attacks, final attacks still need human review.

What it means for monitor testing

For teams building monitors for AI agents, Screwtape offers a way to probe them using realistic transcripts without hand-crafting every scenario. The result that six attacks tuned against proxy monitors also beat a held-out frontier monitor suggests such attacks can transfer between monitors. The low yield, six confirmed attacks from 252 runs, and the need for human review show that the approach remains costly and is not yet a fully automated test. The authors point to a paper with implementation and experiment details.