GPT-6 Astra led coding agents, but finished only 26.4% of tasks

A strong pass rate hid incomplete tasks

Co-Design Bench was published on 26 Oct 2026 by researchers at Alibaba and five universities. It tested ten coding agents on 273 tasks drawn from 37,463 deduplicated queries across 26,314 real user coding sessions.

The tasks asked agents to revise working web applications, sometimes using an attached image, audio clip, video or document. The researchers created 2,331 hidden, expert-authored acceptance criteria. Agents received the user request and a runnable application, but not the criteria.

After each agent completed its work, judges built and ran the application in a browser and assessed every criterion as Pass, Partial or Fail. A task counted as fully finished only when every criterion passed. GPT-6 Astra ranked first, passing 69.9% of criteria per task on average and completing 26.4% of tasks in full. Opus 5 passed 66.1% of criteria and fully finished 18.3% of tasks, while Qwen3.8-Max recorded 65.3% and 16.5%, respectively.

The benchmark focused on difficult, realistic requests

The researchers selected requests using difficulty, verifiability and past failure signals, including reformulations and rollbacks. The benchmark therefore was designed to test demanding work rather than routine coding prompts. Its tasks covered web applications, not backend services, and each was a single fixed request without follow-up questions.

GPT-6 Astra fully finished 41.0% of 100 multimodal editing tasks and 17.9% of 173 repository editing tasks. Across all ten models, repository edits involving three or four change types passed 59.1% of criteria, compared with 65.4% for edits involving one or two types. Of the 173 repository tasks, 166 required more than one change type.

The results also depended on the software harness used to run the agents. Every model was tested in the Claude Code harness, while changing harnesses moved Qwen3.8-27B’s repository-edit score from 56.0% to 74.9%. The researchers used an LLM judge, a separate LLM verifier that challenged positive verdicts, and two experts who checked 200 outputs.

Why reviewing only the diff can miss the problem

Qodo’s analysis is that an agent can complete most of a task while leaving an important requirement untouched. A change that meets nine of ten criteria may look complete in a pull request because the missing tenth requirement produces no incorrect line for a reviewer to inspect. Qodo gives the example of a discount that never reaches the order summary: the necessary code is absent, rather than visibly broken.

The Co-Design Bench paper describes this pattern by saying that “Substantial coverage of individual requirements can coexist with limited complete-task success.” According to Qodo, the benchmark measures agents rather than code review, so the conclusion about review is Qodo’s inference.

Qodo says it reviews pull requests against their linked tickets from systems including Jira, Linear, Azure DevOps, GitHub Issues, GitLab Issues and Monday.com. Its stated approach is to compare the code with the ticket’s intent, acceptance criteria and subtasks, then report missing or incomplete requirements as findings in the pull request. That approach depends on having written requirements: a vague ticket gives reviewers little to verify.

The benchmark’s authors also identify limits, including the difficult task selection and the possibility of judgment error. They write that retained evidence “cannot eliminate judgment error.” An earlier test reported a different result: the paper says Opus 4.8 reached 99.4% feature completion on WebCompass-edit when it could run and fix its own code, while also noting that the two benchmarks used different task pools and grading.