AI21 says post-training pushed Nemotron 3.5 Super past frontier quality
AI21 says it has post-trained NVIDIA Nemotron 3.5 Super VL to outperform Claude Opus 5.5 on Deel’s global payroll benchmark, while costing 50x less per 1M tokens. The result came from a three-week collaboration between AI21’s Forward-Deployed Researchers and Emet, Deel’s AI lab for model evaluation and training.
- AI21 worked with Deel’s Emet AI lab over three weeks.
- SFT and SDPO raised Nemotron’s development-set score to 0.720 / 0.339.
- AI21 reports an 8-point quality lead over Claude Opus 5.5.
- The model was reported as 50x cheaper per 1M tokens than Opus 5.5.
A payroll model built for changing local rules
Deel supports more than 45,000 companies across 150+ countries, according to AI21. Its payroll questions can depend on local tax rules, employment contracts and company policies, all of which change regularly. Answers must also follow Deel’s terminology, policies and style.
AI21 and Deel evaluated the model with EmBench Payroll, Emet’s public payroll benchmark. Each question contains 3, 6 rubric assertions, covering factual details and whether the response fully addresses and frames the situation. The benchmark reports two measures: the average share of assertions satisfied, and the share of questions where every assertion is satisfied.
AI21 says Claude Opus 5.5 met 64.0% of the criteria and fully answered 26.3% of the development questions. The base Nemotron 3.5 Super model scored 51.3% on the criteria measure. Deel was seeking an open model that could provide complete answers without prompting, at a lower cost than frontier models.
Two-stage training improved completeness
AI21’s analysis found that many of the base model’s failed answers were incomplete rather than incorrect. The model often hedged when the law was definite, omitted figures or skipped important context. AI21 says this indicated that the model frequently had the required knowledge but needed to be steered toward more complete answers.
In the first stage, the researchers used near-policy supervised fine-tuning, or SFT. They generated responses with hints covering the model’s role, the expected detail, Deel’s terminology and common distinctions. The hints added no new domain knowledge. The team selected 412 responses that met every rubric assertion and trained the model on them, with the hints removed.
Performance on the development set rose from 0.513 / 0.100 to 0.657 / 0.229, measured by criteria met and all criteria met. AI21 then applied self-distillation policy optimization, or SDPO, using a dataset of 698 questions. Starting from the SFT checkpoint produced the strongest result, 0.720 / 0.339.
Cost and quality gains for a specialised use case
AI21 reports a 20.7-point quality improvement over the base model and an 8-point improvement over Claude Opus 5.5 for the global payroll use case. It also says the resulting model is 50x cheaper than Opus 5.5 when measured by cost per 1M tokens.
The company attributes the result to combining Deel’s expert evaluation criteria with sequential SFT and SDPO training. The small datasets were kept close to the model’s own output distribution, while SDPO generated fresh rollouts and could learn from questions the model still answered incorrectly without requiring a gold answer.
For users, the case study suggests that targeted post-training can improve an open model’s ability to deliver complete, domain-specific answers when expert benchmarks and carefully designed training methods are available. The reported gains apply to Deel’s global payroll use case, rather than establishing a general improvement across tasks.
