Harbor Can Now Run Every Agent Eval Task in Docker Sandboxes

Why agent evals need isolation

Docker describes agent evaluation as a simple loop: give the agent a real task, let it work, and check the result. Along the way the agent installs packages, edits files and runs commands. Docker calls this untrusted, AI-written code, and says a benchmark runs it thousands of times.

There is also a fairness problem. Docker notes that an agent can sometimes earn a passing score by gaming the check instead of doing the work, for example by tampering with tests or overwriting the file that records its score. This is called reward hacking, which Docker describes as a growing concern across the eval community.

Docker says running agent evals therefore needs two things: a framework that runs each task, lets the agent work, checks the result and collects scores and logs, and a clean, isolated environment for every task. Harbor, an open-source eval framework that can take a benchmark such as Terminal-Bench, covers the first. Docker Sandboxes now cover the second.

What Docker Sandboxes add

Docker Sandboxes are microVM environments for running agents such as Claude Code, Codex and Cursor safely, according to the company. Docker lists four benefits:

• Strong isolation by default. Each task runs in a microVM rather than a shared container. A microVM has its own kernel, so what the agent does stays inside its sandbox and cannot reach the host machine, other tasks or anything outside the environment, per Docker.
• Fair, reproducible evals. Configurable network policies let a task download what it needs during setup, then cut the agent off from the internet while it works. Every agent can then be tested under the same conditions.
• Parallel runs without managing machines. With Docker Cloud Sandboxes, teams can run many tasks at once without provisioning anything manually. Docker says it has tested Harbor with 64 concurrent workers on Docker Cloud Sandboxes.
• Local development, cloud scale. The same tasks run locally or in Docker Cloud Sandboxes, so users can build on a laptop and then run tasks in the background and at scale without changing anything.

How to try it

Docker gives these steps. First, install Harbor with Docker Sandboxes using uv tool install "harbor[sbx]". Next, activate Cloud access for your Docker account and create a personal access token, selecting Cloud Sandboxes under access permissions. The token is shown only once, so it needs to be saved.

Then set your Docker ID and token as the environment variables SBX_DOCKER_USERNAME and SBX_DOCKER_PAT, and run a task in Cloud Sandboxes with harbor run -t hello-world/hello-world -a oracle -e sbx.

To run locally instead, install the Docker Sandboxes CLI, run sbx login, and add --ek target=local to the same command. Docker says local use starts for free, with the option to scale to Cloud Sandboxes.

Limits and what it means

The current integration supports single-container tasks that run on CPUs. Docker says GPU and multi-container support are coming soon, without giving a date. Benchmarks that depend on those setups cannot yet use the integration.

For teams that evaluate agents, the pitch is a way to contain untrusted agent code and keep test conditions identical across runs, while moving from a laptop to many parallel workers using the same task definitions. Further details are in the Docker Sandboxes and Harbor documentation.