MLCommons Releases Taxonomy of Five Privacy Risks Specific to AI Agents

Why agents need their own privacy framework

The taxonomy comes from the Privacy and Confidentiality Working Group, part of the MLCommons AI Risk & Reliability (AIRR) effort. AIRR starts from the idea that teams should define the behavior they want from an AI system and then measure how reliably it delivers that behavior. According to MLCommons, deployers need such measurement to manage both risk and cost.

In April, the group published the AI Reliability Map, which sets out the rules a system should follow: functionality, data protections, product safety, frontier safety and psychosocial limits. It also lists the circumstances under which they are tested, namely normal use and under attack. The privacy group is now focusing on how to evaluate an agent handling personal data.

MLCommons argues that the scope of risk changes with agents. For a chatbot working on its own, data protection risks are, in the group's words, for the most part limited to the individual user interacting with it. For an agent, the potential scope is vast. Before anything can be measured, the group says, the community must agree on which privacy risks are specific to agents. The taxonomy draws on expertise from industry, academia and civil society, and on known privacy and security incidents and existing risk management frameworks.

The five areas of risk

• Data ingestion and processing. Agents observe continuously in the background, log every step and carry memory across sessions, so they routinely hold more than a task needs. Redacting names and account numbers does little when free text can reveal deanonymized clues.
• Aggregation, use and sharing. Pieces of data that are harmless apart can be sensitive together. Agents can share or delete things without authorization, depend on third-party tools they cannot vouch for, and can write and run code on the fly. A model may also memorize fragments of personal data from user-consented training datasets.
• Inconsistent privacy practices across agents. One agent may work under one policy and hand off to another agent under a different one. Context can spill into logs, and downstream agents often do not receive deletions, corrections or withdrawn consent.
• Failure of static consent models for runtime agentic behaviour. Consent works as a one-time gate, but agents decide at runtime. Either the agent infers the user's expectations and may be wrong, or it asks every time and users click yes without reading. Because agents do not fully reveal how they reason, users often cannot tell what they agreed to.
• Accountability and governance. It is hard to establish whether the user, the prompt, the model, the tool or another agent caused a failure. According to MLCommons, no standard, privacy-preserving way exists to log agent interactions across organizations, and existing logs and memory stores become valuable targets for hackers.

What comes next

Version 0.1 does not rank the risks. MLCommons notes that capability and context matter: a bank's customer service agent and a personal assistant with access to an inbox face different risks in a different order. The next step is to work with deployers to determine which risks matter most in which kinds of deployment, and which are most universal. The group plans to hold multiple engagements with large-scale deployers, in consultation with AIRR colleagues, then identify indicators for those risks, pilot key benchmarks and develop mitigation best practices. It plans to complete this effort by Q1 2027, while continuing to iterate on the taxonomy.

Some risks can be tested before deployment, such as whether an agent pulls more data than a task needs or passes a deletion request downstream. Others, particularly in accountability and governance, concern the organization around the agent, so a benchmark will not capture them. Sorting out which is which is part of the job, MLCommons says.

What it means for deployers and how to take part

MLCommons encourages industry partners to integrate the taxonomy into AI development and deployment, from pretraining through deployment monitoring. The group says this work will help reduce privacy violations and risks, improve privacy and security protections, reduce potential liabilities and reduce harm to individuals and organizations. The stated goal is measurement tools that show deployers how their agents perform overall and in the areas that matter most in their context.

The group is asking for feedback from industry practitioners, academic researchers, civil society organizations and regulators, and expects the v0.1 to change. The Privacy Working Group meets every other Thursday from 11:30 to 12:30 ET, and interested people can register to become members and contribute.