Cline harness lifts NVIDIA Nemotron 3.5 Super VL to 61.8% on Terminal-Bench 2.1

Benchmark result across four harnesses

Cline tested Nemotron-3.5 Super VL 120B A12B on Terminal-Bench 2.1 and compared its own harness with three other open-source harnesses, labeled A, B and C. According to Cline, the scores were 61.8% for Cline, 53.9% for harness A, 51.7% for harness B and 44.9% for harness C.

Cline reached its score with thinking off, while harnesses A and C used high reasoning or effort settings. The company stresses the limits of the comparison: it shows how much results can vary with the software around the same model, but it does not isolate which Cline features account for the difference or measure how much compute they save.

Cline defines an agent harness as the software around the model: the instructions it receives, the tools it can use, and the checks that guide its work. It says it has been refining those parts of Cline for open-weight models, including Nemotron 3.5 Super VL.

What Cline changed in its harness

Cline points to three areas of work.

• Planning before coding: Cline prompts the model to understand the task, choose an approach and only then start making changes. The company thinks this can help the model spend less effort deciding what to do next, but says it has not measured token savings or isolated the effect of the planning instruction.
• Stall detection: Cline's stall detector can end an attempt that keeps repeating an action and record a structured event with the error class, whether it is recoverable, and the iteration number. Cline says this makes failed runs easier to investigate.
• Behavior checks: The agent verifies the behavior a task depends on rather than stopping at a surface-level test.

Cline gives two examples from the evaluation. On chess-best-move, Cline parsed the board into FEN notation and ran an exhaustive checkmate search, finding two mates in one, e2e4 (Qe4#) and g2g4 (g4#), in a run spanning 58 log lines. Harness A made 29 tool calls and reopened the board image at least six times before timing out.

On headless-terminal, harnesses A and B both ran real tests. Cline completed its checks in five commands, using an itemized checklist that covered both whether Ctrl-C delivered a signal and whether the shell survived afterward. According to Cline, the narrower checks in the other runs missed the behavior that eventually broke.

An internal Nemotron agent in Slack

Beyond the benchmark, Cline has built an internal agent around Nemotron that works with its team in Slack and runs Cline under the hood. It can spin up sandboxes and containers, and call more advanced models when a task needs them. The team uses it to triage internal tasks and to investigate and fix issues reported by users.

One example involved a Windows diagnostics export failure. A teammate asked the agent to investigate a report in which Cline tried to create an existing Downloads folder, hit an EEXIST error, and failed to write the diagnostic report. Cline describes this as the kind of work it wants Nemotron to help with: taking a user report, tracing the failure, and giving the team a useful next step.

What it means for open-weight model users

The result suggests that the harness around an open-weight model can shift benchmark outcomes noticeably, even when the model stays the same. Cline frames the benchmark as a way to compare performance and the internal agent as a way to see how the model handles real team work.

The company is open about what remains unproven, including token savings and the individual effect of its planning instruction. It says it will keep refining Cline through benchmark runs and the issues its team handles every day.