Apple Study: Elaborate Harnesses Add Little for Strong ML Engineering Agents
A new paper published on Apple's Machine Learning Research site finds that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses offer no advantage over a single session of a minimal-harness coding agent. The authors conclude that the backbone model is the primary driver of performance in autonomous machine learning engineering.
- Under equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses gave no advantage over a single session of a minimal-harness coding agent.
- The minimal agent gets direct access to the execution environment through read, write and bash primitives.
- Ablation studies led the authors to argue that the machinery layers become redundant in the coding agent setting.
- The authors conclude that hand-crafting harnesses around strong models yields poor returns for current MLE benchmarks.
- The paper is by Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé, with EPFL affiliations.
What the researchers tested
The paper, titled "How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?", is by Kirill Brilliantov, Alejandro Hernández-Cano and Emmanuel Abbé. Brilliantov and Hernández-Cano contributed equally, and the affiliations listed are EPFL, with a note that Hernández-Cano's work was done while at Apple.
The study looks at autonomous machine learning engineering (MLE) agents, which according to the authors have made significant progress on public leaderboards. Many of these agents, they write, are built on increasingly elaborate machinery, such as multi-agent orchestrators and dedicated retrieval subagents. The authors say this is often motivated by progress stagnating over long-horizon cycles and by the limited primitives available to large language models.
A minimal baseline against elaborate harnesses
The researchers point to a gap in the field. While harnesses keep expanding, they say, simpler but improved coding agents have received little attention. In this setup, the LLM has direct access to the execution environment through read, write and bash primitives.
Using such a minimal-harness coding agent as a baseline, the authors compared it with open-source state-of-the-art harnesses. Both used the same frontier LLM backbone and an equal time budget. The baseline ran as a single session. According to the paper, the elaborate harnesses provided no advantages over it.
The authors say this points to the backbone model as the primary driver for performance.
Ablations and the authors' conclusion
Beyond the head-to-head comparison, the paper reports a series of large-scale systematic ablation studies. From these, the authors argue that the machinery layers become redundant in the coding agent setting.
Their conclusion is direct: the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks. The qualifier matters, since the claim is tied to current MLE benchmarks and to strong frontier models, not to every agent task or every model.
What it means for agent builders
For teams building autonomous ML engineering agents, the findings suggest that investment in multi-agent orchestration and retrieval subagents may deliver less than expected when a strong model already has direct access to the execution environment. The authors' results point toward choosing a capable backbone and keeping the surrounding system simple, at least for the benchmarks they studied.
The paper is classified under Methods and Algorithms research on Apple's site. The source text does not give specific benchmark scores, model names or datasets, so the size of any differences between the setups is not stated here.
