Patronus AI Details In-House Post-Training Stack and First Results on Qwen3.8-27B

What the first experiments showed

Patronus AI tested two recipes: supervised fine-tuning (SFT) followed by reinforcement learning (RL), and RL alone. The base model was Qwen3.8-27B, which Alibaba has already post-trained. It was trained on two task families: computer use in the GitHub application, and Unreal Engine development across five separate projects.

On held-out GitHub tasks, the starting model solved 35.6%. SFT on 41 trajectories added 5.4 points, which the company says is too small to separate from noise. RL then added 6.8 points by step 40, for 47.8%, a 12.2-point jump. RL alone reached 45.4% at step 40 (+9.8). When training continued to step 60, RL alone reached 49.2% (+13.6).

On Unreal Engine tasks, the starting model already solved 64.3%, leaving less room to improve. SFT then RL reached 71.5% (+7.2), and RL alone reached 73.1% (+8.8). According to Patronus, failures fell by roughly one-fifth with SFT then RL and about a quarter with RL alone. Both task families fixed about one-fifth of the starting model's failures.

Caveats the company points out

The task sets were small: 46 held-out GitHub tasks and 54 held-out Unreal tasks. Patronus says the gains are larger than the noise, so the improvement over the starting model is real. The gap between SFT then RL and RL alone is roughly 2 percentage points, which it says is within the noise, so it cannot say which method is more effective.

The company notes that RL alone performing about as well as SFT then RL matches research from the wider community, citing DeepSWE and SWE-RL, which found that RL alone helped coding agents. Patronus also says its setup mildly favors the SFT recipe, because Kimi-K3 served both as the expert model used to select held-out tasks and as the author of the SFT data.

The checkpoints were fixed before evaluation and are not the best-scoring ones. The source text also describes a check of 50 passing runs to confirm the model was not reward hacking, but the excerpt ends before the findings.

How the stack works

Patronus describes five stages that loop from training to a shared leaderboard. A command starts SFT or RL, checkpoints upload and merge into full models, and a single click turns a checkpoint into an OpenAI-compatible vLLM endpoint, one model per node. The evaluation platform then runs the held-out benchmark suite under one shared preset, and scores land on a leaderboard comparing base and post-trained models with the same preset, dataset hashes and environments.

The company calls the leaderboard the least flashy stage but the one that matters most to its research team, because it holds test tasks, attempts and infrastructure failure rules constant. The platform snapshots code, records the git commit and any uncommitted changes, and launches jobs through Slurm. A web dashboard supports cancellation and sweeps across up to 50 jobs and links each run to its Weights & Biases page. Patronus says the stages are currently joined by scripts, and it plans to keep automating them until a single training command can produce comparable numbers.

For SFT, the team used NVIDIA NeMo Automodel with LoRA on the language model and context parallelism for long trajectories. For RL, it used NeMo-RL with GRPO, with rollouts generated in vLLM sharing GPUs with training. Tasks run outside the GPU cluster on the company's own environment service, where each attempt gets a sandbox with the real application, the agent's tools and a grader.

Task design and evaluation details

Near copies of a task, such as reworded variants, are grouped and placed entirely in or out of the training set, to stop the model from memorizing training tasks. Held-out tasks are spread across projects and difficulty levels and limited to ones the expert model solved at least once. SFT used successful Kimi-K3 trajectories, with at most 4 per task and none more than twice as long as the shortest solution.

For RL, the reward is 1 when a task's grader passes. Because models learn only when some attempts succeed and others fail, Patronus tuned task difficulty to the model. That produced 175 GitHub tasks for SFT then RL and 139 for RL alone, plus 53 Unreal tasks for both recipes. Each held-out task is run eight times, and a few percent of attempts fail because the environment breaks, which the company says changes no gain by more than a percentage point.

What it means

The results are early and based on small samples, but Patronus says they justified building its own stack as its task pipelines scaled. The practical aim is reproducibility: letting anyone on the team trust a number without rerunning experiments.