Google's Project Chrysalis shows a self-patching AI agent fixing a simulated outage

The problem: advisory copilots fall short in live outages

Anant Nawalgaria, a Group Product Manager and AI Engineer at Google, writes on the Google Developers Blog that generative AI copilots have sped up code authoring and log summarization, but AI agents still struggle to reliably remediate live production outages. Tightly coupled microservices can fail in cascades that, in his words, no static runbook or threshold alert can anticipate.

The post uses a Black Friday flash sale as its example. A burst of checkout requests hits cold inventory keys, Google Cloud Memorystore (Redis) evicts them under Least Recently Used limits, and the cache hit ratio collapses to zero. Traffic then floods Google Cloud SQL, the PostgreSQL connection pool saturates, and the service starts returning HTTP 500 errors.

Google also describes what it calls the "3:00 AM Panic Reflex": both human responders and naive LLMs instinctively restart the database connection pool. During a live stampede, the post says, that locks the database kernel, spikes tail latency nearly tenfold and pushes error rates toward total outage.

What a self-evolving agent does

Google defines Self-Evolving AI as a system that can do five things: diagnose distributed root causes across cache and database tiers, self-patch, self-critique, remember, and re-architect. Conventional agents are frozen at deployment with fixed system prompts and pre-registered tools. The Chrysalis agent instead extends those baseline capabilities at runtime.

• Self-patching: when the agent finds that database starvation stems from an unthrottled ingress surge and has no traffic-shaping tool, it writes an in-memory token-bucket rate limiter, validates its Abstract Syntax Tree, signs it with Cloud KMS and hot-mounts it into the API Gateway.
• Self-critique: after the outage, it evaluates the telemetry change caused by each action, penalizing the database restart because it worsened latency and rewarding the rate limiter for eliminating server errors.
• Memory and rewiring: it writes a signed operational principle into vector memory that blocks database restarts during cache stampedes, then commissions a permanent Traffic Shaping Specialist Agent under the supervisor agent.

Three research pillars behind the design

The architecture combines three research efforts to address what the post calls the SRE Trilemma: balancing exploration safety, structural adaptability and multi-step credit assignment.

• Dream-RSI, serverless counterfactual simulation: production request traces are exported to Google BigQuery, where the engine replays hundreds of candidate token-bucket configurations in sub-millisecond vectorized memory sweeps to find Pareto-optimal policies offline. This avoids testing unvetted parameters on live traffic.
• APEX, tri-level co-evolution: evolution is split into three governed timescales. Layer 1 hotpatches with in-memory middleware tools, Layer 2 distills Cloud KMS-signed lessons into Agent Platform Vector Search, and Layer 3 adds permanent specialist sub-agents to the multi-agent Directed Acyclic Graph.
• AgentEvolver, step-level Shapley attribution: it computes marginal Shapley credit for each execution span using pre- and post-intervention changes in P99 latency, HTTP 5xx rate and database pool saturation, separating helpful patches from harmful ones.

Governance: not an unconstrained black box

Google stresses that the system does not operate as an unconstrained "black box". Every autonomous action is bounded by a defense-in-depth stack on Google Cloud. Model Armor sanitizes incoming telemetry against prompt injections and data leaks. Cloud KMS and Binary Authorization middleware sign synthesized patches and block unverified code from mounting. An Open Policy Agent Policy Controller enforces SLA guardrails meant to prevent self-inflicted outages. Dual-Custody Human-in-the-Loop governance requires explicit SRE sign-off before any permanent change to the agent hierarchy.

Reported results and what they mean

Google built the architecture as an interactive application on the Gemini platform using the Agent Development Kit (ADK). In the first demo phase, a 5-request warm baseline is followed by a 50-request flash-sale surge across 15 cold inventory keys. The Memorystore hit ratio falls from 100% to 0.0%, the Cloud SQL pool reaches 100% saturation (2/2 active connections locked), P99 latency spikes to 312.8 to 519.8 ms, and HTTP 500 errors reach 72.7% to 80.0%, with 40 failed checkouts.

In the second phase, the ADK Supervisor Router Agent, which Google says is powered by Gemini 4 Argon, is woken by telemetry alerts after Model Armor inspects the payload. The Layer 1 synthesizer then generates a rate limiter with a 10-token capacity and 40.0 req/s replenishment. The source text provided ends before describing the rest of the walkthrough.

Across the simulated outages, Google reports that time-to-first-mitigation fell from 15 to 45 minutes to 1.8 seconds, HTTP 5xx error rates went from 72.7% to 80.0% down to 0.0%, and P99 latency dropped from 519.8 ms to 18.2 ms. It also says 400 counterfactual traffic policies were evaluated in BigQuery in about 0.5 ms.

These figures come from simulations and a company research reference architecture, not from production deployments. For SRE and platform teams, the post is a blueprint for pairing agent autonomy with signing, policy checks and human approval, rather than a finished product.