AI21 and Alice post-train Nemotron 3.5 Super VL to beat GPT-6.1 Sol on attack resilience

Training agents to resist attacks without losing usefulness

Alice builds reinforcement learning environments for post-training. According to the post, they draw on real-world intelligence about threat actor behavior and input from more than 150 staff security researchers and expert network members. Alice gave AI21 a set of RL gyms for training agents to defend against security threats in enterprise settings, including tools such as Slack and GitHub.

One target is indirect prompt injection: malicious instructions planted in information an agent reads with tools. AI21 gives the example of an agent asked to send a password reset reminder to employees on Slack, where the reset link has been corrupted and could expose login credentials. A trained agent should detect the problem and block it, while still notifying employees to reset their passwords.

The aim is agents that score highly on both resistance to attacks and task completion, so that neither comes at the expense of the other.

Two challenges and the training recipe

AI21 describes two core challenges. The first was improving attack resilience without hurting performance on regular benign tasks. An agent that simply stops working after spotting a threat defends the company but is not useful. The second was data: the training set held several hundred examples, so the team had to avoid overfitting.

Each Alice gym is packaged as a Harbor task with its own container image holding a simulated workspace. For every rollout, the AI21 Training Platform launches a fresh, isolated sandbox, and anything the agent does in response to an attack stays inside it. The same setup was used to generate trajectories for supervised fine-tuning (SFT), to produce live rollouts for GRPO and to evaluate every checkpoint.

The team first ran an SFT stage because, in the initial evaluation, rewards of Nemotron 3.5 Super rollouts rarely varied within a group, and GRPO learns only when they do. After SFT, the share of training tasks with non-zero reward standard deviation rose from 35% to 66%.

The SFT data came from successful rollouts of several open models. To avoid importing their long, sometimes repetitive reasoning and their writing style, the team had Nemotron 3.5 Super rewrite the reasoning in each trajectory in its own words, leaving every action unchanged. According to AI21, the original rollouts were ×1.99 in perplexity relative to the base model's own, a policy trained on them drifted to ×1.40, and one trained on rephrased data stayed at ×1.23. The rephrased-data checkpoint matched the quality of the original while generating far fewer tokens.

Reward shaping in online RL

In the second phase, the team ran online RL with GRPO. Vanilla GRPO subtracts the group's mean reward and then divides by the group's standard deviation. AI21 found this a poor fit for Alice's gyms, which use dense, continuous rewards. Dividing by the spread discards the size of the reward gap: groups with rewards [0.49, 0.51] and [0, 1] both end up with advantages [−1, +1]. Small, possibly noisy differences, such as a slight shift in an LLM judge's score, can then be amplified into full-strength gradients.

The team therefore dropped the standard-deviation normalization and kept only the group-mean baseline. According to the post's figure description, removing it lifted reward above the repetition penalty alone, and adding a length penalty kept reasoning from growing while reaching the highest reward. The source text is cut off while introducing another reward modification.

What it means

AI21 argues the work shows that specialized open models can surpass frontier-level quality at a fraction of the cost. It says Nemotron 3.5 Super VL moved from quality and cost comparable to GPT-6.1 Luna to beating GPT-6.1 Sol. As enterprises give agents broad access to sensitive information and employees, the company says it is essential that those agents can spot and stop attacks, and that its results show high attack resilience can coexist with task completion.