Harness-Zero study trains an agent harness into a small model

How Harness-Zero moved a harness into a model

The paper, called Harness-Zero, comes from Haoran Ye and colleagues at Peking University. It deals with harness distillation, which means training a model to reproduce behavior that used to come from the harness around it.

The team first built a specialized harness for a small model, Qwen3.5-9B. An automated loop run with Kimi K3 analyzed where the small model kept failing on training tasks and added components to fix those failures: custom tools, middleware rules that block risky actions, skills files and memory. The researchers called the result the "evolved harness".

That harness was then used to generate training data. The small model worked through training tasks inside a minimal harness while a stronger model, GPT-5.6 Sol, reviewed every step the small model proposed. Sol used a copy of the evolved harness to judge each step. Good steps passed through unchanged, and where the small model and the guidance disagreed, Sol made the smallest fix needed before the step ran. The corrected runs became the fine-tuning data, and the evolved harness, the reviewer and everything else were then removed.

Results and limits

The team compared the evolved harness and the distilled model on spreadsheet editing, multi-app tool use and chemistry. According to Arize's summary, the distilled model's average task success rose 21 points versus its untrained predecessor. That is more than the untrained model gained with the full evolved harness attached.

There are caveats. The distilled model still ran inside a minimal loop with 1 Bash tool, and the authors say their method narrows what a harness has to provide rather than eliminating it. In chemistry, where the task is predicting which molecules combine to make a target compound, the distilled model trailed the harnessed version by 8 points. The harness there supplied chemistry knowledge, candidate-generation logic and a molecule validation tool.

The authors' explanation is that step-by-step procedures, such as inspecting before editing and verifying before finishing, are much easier to train in than deep domain knowledge. Context management did not fully transfer either.

The type of training data mattered. Fine-tuning on the stronger model's own successful runs left the small model's score where it started, as did fine-tuning on runs completed inside the evolved harness. Giving the reviewer the correct answers produced nearly perfect runs but a model that barely improved, because it over-fit. Only the corrected runs, where each fix was still within the small model's reach, raised task success by 21 points.

Why models keep absorbing harness logic

The Arize post ties the result to Rich Sutton's 2019 essay The Bitter Lesson, which looked back over 70 years of AI research and found that built-in human knowledge helped in the short term but eventually lost to general methods that improve as computation gets cheaper. A harness is that kind of built-in knowledge, the post argues, and model training is the learning that absorbs it.

Others have made similar predictions. According to the post, Jesus Rodriguez at TheSequence wrote in January that hand-coded scaffolding keeps losing to capabilities models learn directly. In February, Boris Cherny, who created Claude Code, told Y Combinator that scaffolding might buy 10% to 20% on a task, and then "the gain is wiped out with the next model." In March, Vivek Trivedy at LangChain predicted that planning, self-verification and long-running coherence would move from the harness into the model.

Arize notes that most of these people build harnesses for a living. They are not saying harnesses go away, only that they must evolve quickly.

What still belongs in the harness

Context management remains a harness job. In ARC Prize's results for ARC-AGI-3, Gemini 3.8 Flash scores 3.4x higher with one harness than with another. The better harness keeps the model's reasoning state between requests and compacts long conversations.

Anthropic's Opus 5.5 prompting guide, released the week of the post, points both ways. It suggests deleting "think carefully" instructions because the model now sets its own amount of thinking, and advises re-testing scaffolding built for charts and screenshots. At the same time, it adds harness pieces: a system prompt paragraph that keeps unattended agents from stopping early, a reminder sent when a long run goes quiet, and elapsed-time budgets for multi-agent setups.

Automating harness updates carries its own risk. Google's RRSI paper found that unconstrained automatic harness evolution overfits. In its tests, the unconstrained version scored highest of any variant on the tasks it evolved against, but landed within a point of the original, unevolved harness on benchmarks it had not seen. RRSI's fix, as Arize describes it, is largely bookkeeping: for every candidate change it records which component the change touches, what it was meant to fix, the diff, and how score and cost moved.

What it means for agent builders

Arize's conclusion is that teams still need a harness, but should expect to throw it away and rebuild it as models improve. The parts that survive are those tied to an application's own tools, data, users and environment. Generalizable procedures are candidates for training into the model, while domain knowledge and context management are likelier to stay in the harness. The practical advice is to automate the rebuilding process, with safeguards against overfitting, rather than treat it as a manual chore with every model generation.