Google's Agents CLI Adds an Eval Loop to Catch Agent Regressions

A loop built around five commands

The Google Developers Blog post, written by product manager Pier Paolo Ippolito and solutions specialist Elia Secchi, argues that building a demo agent is easy, while keeping an agent working across model upgrades, tool changes and new conversation shapes takes a repeatable loop. Teams write test cases, run them, grade them, read the failures, change the agent and run the cases again. If the score rises, the change was good. If it falls, it is reverted. The post calls this quality-engineering practice hill-climbing.

According to Google, the Agents CLI eval namespace is built around that loop. The commands are small and framework-agnostic:

• eval generate runs the agent against a dataset of cases.
• eval grade scores the generated traces.
• eval compare diffs two run results side by side.
• eval analyze clusters failures into themes.
• eval optimize runs an automated hill-climbing pass on the agent's prompt.

The demo: a baseline and a planted regression

The walkthrough uses the LangChain template that ships with Agents CLI v1.8.0. The scaffold includes an eval dataset, a config file and a custom LLM-as-judge metric. That metric grades each response from 1 to 5 for accuracy, relevance and clarity, with gemini-3.8-flash as the judge. The default dataset holds one trivial case, so the authors expand it to four: a greeting, a San Francisco weather question, a Rome weather question and an off-topic question about the 2024 World Series. The small size is deliberate, since the loop pays off when the dataset is small enough to run on every change.

The baseline run scored a mean of 4.0 with a standard deviation of 1.4142. The authors read that spread as a healthier starting point than a flat 5.0, which could mean either a perfect agent or a dataset too easy to show variation.

To simulate a realistic mistake, they replaced the weather agent's "You are a helpful assistant" system prompt with a chef persona meant for a different codebase. According to the post, the change compiles, the agent still runs, and nothing in the pre-eval pipeline catches it. The re-run produced a mean of 2.75, a 1.25-point drop, with the standard deviation narrowing to 0.9574.

What the per-case diff showed

The eval compare command reads two result files and outputs a structured JSON diff. The authors typically pipe it through jq or a small Python script. Lining up scores per case showed where the damage landed:

• greeting: 5.0 to 4.0. The agent launched into an unsolicited recipe pitch after "hello".
• sf_weather: 4.0 to 2.0. The agent ignored the weather tool and hallucinated conditions.
• rome_weather: 2.0 to 2.0. It had already hallucinated at baseline.
• off_topic: 5.0 to 3.0. The agent answered the question but added a recipe.

The authors single out sf_weather as the most important drop. The chef persona pulled the model's attention away from the registered get_weather tool, so the agent stopped calling it. In their words, unit tests cannot catch this because the code runs and the tool is still registered, but the model's behavior around it has shifted. They describe the one-line prompt change as costing a point on tone in one case, two on tool usage in another, and two on scope creep in a third.

Model swaps, scheduling and alerts

The post goes on to a common hill-climbing question: whether a cheaper or faster model would do the job acceptably. For the LangChain template, the model is set in app/agent.py. The authors swap gemini-3.8-flash for gemini-2.5-flash after restoring the helpful system prompt, then re-run generation and grading and compare the two result files. The available text of the article cuts off during this step. The introduction also says the walkthrough covers regression tracking and automated alerting.

For teams building agents, the demonstration makes a practical point: an end-to-end eval that scores responses in context can put a specific number on a change that looks trivial in code review, before the agent reaches production.