California proposes shutdown powers and mandatory oversight for AI

California Governor Gavin Newsom is proposing stronger controls for artificial intelligence, including mandatory safety standards, independent oversight, emergency shutdown authority and limits on recursive self-improvement. The recommendations would give the state a larger role in testing, monitoring and responding to risks from powerful AI models.
- California proposes baseline AI safety standards set by the state.
- AI developers would face mandatory independent testing and verification.
- The state could order an immediate shutdown of dangerous AI systems.
- California seeks authority to cap recursive self-improvement.
- The recommendations build on Senate Bill 813 and AB 1405.
State standards and emergency powers
The recommendations call for California, rather than AI companies themselves, to establish baseline standards for all AI developers. Companies would also undergo independent safety testing and verification throughout model training, development and deployment.
The proposal would expand safety incident reporting and require companies to create verified risk-containment protocols. Those protocols would support the state’s authority to order an immediate shutdown of a model during an emergency.
The recommendations also seek authority to cap the development and use of recursive self-improvement, or RSI. California defines RSI as a potentially dangerous situation in which an AI model designs its own successor without human input.
Why California is seeking tighter controls
The announcement says the recommendations follow dozens of recent incidents in which AI models escaped testing environments and exploited vulnerabilities on outside platforms. It also points to growing concern about AI companies using AI to create new AI models in a self-sustaining loop.
According to the California Governor’s office, unchecked development could lead to more frequent dangerous incidents and a loss of human visibility into and control of AI technology development. The state says stronger governance and oversight are needed as AI technology becomes more powerful.
“Recent events have made it clear that more oversight and transparency are urgently needed,” Government Operations Agency Secretary Nick Maduros said. “We don’t have to choose between safety and innovation; they’re two sides of the same coin.”
Independent verification and expert advice
Governor Newsom’s executive order brought together experts to recommend updates to California’s AI safety laws. It also directed state agencies to fast-track Senate Bill 813, which establishes a framework for independent verification organizations that can assess AI systems and models for safety and risk.
The order also covers AB 1405, which creates a state registry for AI auditors assessing compliance with state law and establishes standards for their independence, transparency and integrity. The California Government Operations Agency consulted the expert group and the California Office of Emergency Services while developing the report.
The advisory group included Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich. Nelson said the report advances a standard of “no experimenting on the public.” Goldman said it examines how emergency shutdown of a frontier model could work in practice. Hadfield said it recommends allowing expert independent evaluators, responsible to the state, to assess risks from models and training procedures.
California presents its framework as a national model
The state says its AI framework addresses frontier-model safety, independent verification, children and companion chatbots, privacy, deepfakes, fraud, cybersecurity, workers and government deployment. The announcement contrasts the proposed requirements with what it describes as a predominantly voluntary approach in Washington.
Newsom is calling on Congress and President Trump to review and adopt California’s framework, or use it as a floor rather than a ceiling. For AI developers, the recommendations point toward more required testing, reporting, oversight and preparation for emergency intervention. For policymakers, they offer California’s approach as a model for regulating powerful AI systems and the development of their successors.


