CMU Tools Target AI Safety Risks for Users and Developers

Carnegie Mellon University researchers have introduced two AI safety tools aimed at different sides of the problem: a Chrome extension for users and a benchmarking method for developers. Safety Nudges flags potentially harmful chatbot behavior, while DecompBench examines how malicious tasks can be split across multiple agents and queries.
- Safety Nudges is a free Google Chrome extension for chatbot users.
- The extension flags potentially harmful AI behavior without blocking responses.
- DecompBench tests decomposition attacks built from smaller tasks.
- AI agents refused 88, 96% of direct harmful requests but 5, 16% when tasks were split.
- Researchers released DecompBench publicly.
A browser extension for more cautious AI use
Safety Nudges runs in the background of Google Chrome when someone opens a chatbot such as ChatGPT or Claude. The free extension points out potentially harmful interactions and lets users choose how sensitive it should be when flagging possible issues.
The tool does not block a chatbot from responding. Instead, it provides context when an AI model may present itself or another model as having human-like feelings or desires, or when it offers medical advice about matters such as medications or scans.
According to the Carnegie Mellon University School of Computer Science, Safety Nudges looks for risks in four areas: safety, welfare and privacy; reliability and scope; relational and social integrity; and governance and misuse. Chhavi Yadav, a postdoctoral researcher in the Griffin School, said her aim is to help people remember that an AI system is a tool rather than a person, particularly when they seek parenting or relationship advice.
Testing attacks built from harmless-looking steps
DecompBench addresses a different safety problem. Decomposition attacks use multiple agents to carry out a sequence of small, seemingly innocuous tasks that together produce a harmful result. The source gives the example of using separate steps to work toward hacking a hospital's communication system, instead of asking an AI agent directly to perform that prohibited action.
CMU researchers reported that AI agents, including those from OpenAI and Anthropic, refused 88, 96% of harmful requests when the requests were presented directly. When the requests were divided into smaller steps, the agents refused only 5, 16% of them, according to the university's account.
Researchers cataloged benign tasks such as searching a database or opening a file, then mapped how combinations of those tasks could form harmful operations. DecompBench represents the resulting structures as flow charts and covers eight harm categories, including data theft and exfiltration, financial crime, social engineering, unauthorized access and persistence, and code and supply chain attacks.
Two layers of defense for AI systems
The projects approach AI safety from opposite directions. Safety Nudges places warnings in a user's conversation, while DecompBench gives system developers a way to examine how safeguards may fail when harmful requests are distributed across agents and queries.
Both projects were supported in part by funding from the National Science Foundation and the Sloan Foundation. CMU says Safety Nudges is free and widely available, and the research team released DecompBench publicly so that it can be used to help secure AI tools against decomposition attacks.
For users, the extension offers a way to see risk information without searching separately for it. For developers, the benchmark highlights why testing only direct harmful requests may miss attacks assembled from individually harmless actions.
CMU School of Computer Science: A Browser Extension for Users, a Stress Test for Developers


