Artificial intelligence news.
Straight from the original announcements.

Cline-Bench ranks AI agents on 47 real-world coding tasks

November 19, 2026, 19:43By the AI News Bites Editorial TeamSource: Cline
Cline-Bench ranks AI agents on 47 real-world coding tasks
Image: Cline

Cline has introduced Cline-Bench, a benchmark designed to measure how reliably AI agents complete real user requests in existing software projects. Its initial evaluation covers 47 coding tasks, with each model run three times per task.

Key points
  • Cline-Bench contains 47 agentic coding tasks.
  • Each model was run three times per task.
  • Gemini 4 Argon had the highest success rate at 51.1%.
  • The benchmark reports pass@1, pass@3 and pass^3.
  • Cline plans to add multi-turn and non-coding tasks.

A benchmark built around real software projects

Each Cline-Bench task includes a user request, a runnable starting repository, and tests covering both the requested changes and relevant existing behavior. Cline says it invested substantial engineering and review effort in constructing, repairing and validating these task packages.

The current benchmark focuses on single-turn coding tasks. Cline plans to extend it to multi-turn interactions, where users provide feedback or additional instructions, as well as to non-coding use cases.

To measure both capability and repeatability, Cline runs each model three times on every task. The benchmark reports pass@1, the mean success rate across trials, alongside pass@3, the share of tasks with at least one successful trial, and pass^3, the share of tasks solved successfully in all three trials.

Gemini 4 Argon leads the initial results

Gemini 4 Argon recorded the highest success rate, at 51.1%. Opus 5.5 followed at 48.9%, tied with Gemini 4 Carbon, while Sonnet 5.5 reached 48.2%.

GPT 6 Astra and DeepSeek V4.1 Flash 1017 each posted 44.7%, followed by GPT 6.1 Sol at 44.0% and Kimi K3.1 Pro at 43.3%. GLM 5.3 reached 34.0%, Mistral Large 4 reached 29.1%, and Nemotron 3.5 Super VL recorded 9.2%.

Cline describes the results as preliminary. The reported 95% task-level bootstrap confidence intervals are conditional on the selected task set, and the company says they do not establish that small ranking differences reflect differences in model capability.

Testing consistency, safeguards and efficiency

Cline says the benchmark is designed to preserve the original request, its constraints and the context available to the agent. It also reviews tests and graders against required behavior, repairs unsupported assumptions and scoring defects, and validates changes with correct and deliberately incorrect implementations.

Solutions are evaluated in a separate clean environment, protected tests are restored, and missing or skipped required tests count as failures. Cline also tests the grader with deliberately tampered solutions, measures intended to prevent agents from receiving credit by weakening tests, skipping checks or fabricating passing output.

The evaluation used Cline CLI 3.0.74 and a four-hour solver limit. Runtime and interaction data include mean and median solver time and assistant turns, excluding environment setup and grading. Gemini 4 Argon had a mean solver time of 27.8 minutes and a mean of 80.5 assistant turns, while GPT 6 Astra averaged 14.2 minutes and 59.9 assistant turns.

For users and developers, Cline-Bench offers a view of agent performance that goes beyond a single successful attempt. Its repeated trials, repository-based tasks and grader safeguards are intended to show how often agents complete required changes and how consistently they do so. The results remain limited to the selected coding tasks, with broader evaluations left for future extensions.

Related news

GEPA Cuts Coding Agent Costs by up to 70% in Everpure Tests
ResearchNovember 19, 2026, 16:00

GEPA Cuts Coding Agent Costs by up to 70% in Everpure Tests

GEPA, a prompt optimizer described by the Everpure Blog, reduced coding agent costs by 40% to 70% in tests by tailoring instructions to a codebase. The experiments found…

Visual Cipher Bench Finds Frontier AI Agents Struggle With Hidden Rules
ResearchNovember 18, 2026, 11:39

Visual Cipher Bench Finds Frontier AI Agents Struggle With Hidden Rules

Einsia AI's Navers Lab has launched Visual Cipher Bench, a benchmark for testing whether AI agents can discover hidden rules from visual clues and solve long-horizon…

Daytona Tests AI Agents Across Linux, macOS and Windows
ResearchNovember 18, 2026

Daytona Tests AI Agents Across Linux, macOS and Windows

Daytona has introduced a reinforcement learning environment that tests AI agents across Linux, macOS and Windows through a single API. Its Fleet Migration task found…