Terminal-Bench 4.1 tightens grading for AI terminal agents

A benchmark for real terminal work

Terminal-Bench measures whether an AI agent can use a computer terminal to complete software engineering and systems tasks. These include writing and debugging code, configuring environments and recovering from failure.

The 66 tasks in version 4.1 cover seven domains: software, science, machine learning, operations, security, hardware and media. The benchmark is hosted by Harbor and the Laude Institute. Each task contains an instruction, an environment and a verifier. The environment provides the files, tools and resource limits, while the verifier examines the result and returns a reward that contributes to the reported score.

Terminal-Bench is used to report frontier agent performance and is commonly cited across model cards. Its 4.1 leaderboard lists 23 model and agent configurations, with resolution rates ranging from 12.1% to 65.8%. Seven configurations resolve more than half of the tasks. Opus 5.5 ranks first at 65.8%, followed by Sonnet 5.5 at 63.9% and GPT-6.1 Sol at 60.9%.

RewardKit Workers strengthen verifier isolation

Snorkel AI, which contributes to Terminal-Bench as a task author, data partner and supporter through the Open Benchmarks Grants program, says version 4.1 focuses on how reliably the benchmark judges completed work.

Verifiers can execute agent-supplied artifacts inside the verifier container. If submitted code can reach the grader or its reward output, it may interfere with the grading process instead of completing the task. Version 4.1 uses RewardKit Workers in affected verifiers. RewardKit is an optional package from the Harbor team that simplifies verifier creation.

A Worker runs submitted code as an unprivileged user, prevents it from reading the tests or writing the reward, and stops processes it started before returning a result. Snorkel AI says this strengthens the grading boundary, but does not replace broader sandbox and resource controls. The release also includes verifier fixes intended to improve grading correctness.

The benchmark addresses other sources of misleading scores through task histories, public verifiers and open issue triage. Its earlier 4.0 release reduced the task set from 74 to 66, removed saturated or problematic tasks and fixed 19 tasks. It also recalibrated resource limits and gave every task an eight-hour agent timeout.

Pinned releases and a CPU-only dataset

Terminal-Bench 4.1 adds prebuilt release images pinned by digest for task environments, verifiers and Compose services. It also pins container images and locks task dependencies, helping researchers reproduce runs against defined versions.

The release includes a CPU-only dataset of 63 tasks. It was published with the same release tag and task revisions as the full dataset. No tasks were added or removed, but all 66 tasks were modified, so results should identify the exact benchmark release.

When the 4.0 leaderboard trials were re-graded with the 4.1 verifiers, about 1% of rewards changed. Leaderboard scores remained within previously reported confidence intervals. For users of the benchmark, the release therefore changes the grading and reproducibility infrastructure without presenting a new task set. For researchers comparing agent results, the exact version, task revisions and verifier configuration now matter directly when interpreting scores.