National Security Experts Call for More AI Evaluation Science After Survey

A survey of national security practitioners

The Institute for Security and Technology, with support from the Future of Life Institute (FLI), released what it calls a first-of-its-kind survey of how national security professionals perceive the implications of artificial intelligence. More than 100 respondents, from former Under Secretaries to retired flag officers, gave their views on the impact of powerful new AI models on military capabilities and decision making, cybersecurity, and weapons of mass destruction.

According to IST, the AI and the Future of National Security survey produced 58 findings, 31 quantitative and 27 qualitative. To mark the release, IST gathered 24 experts from industry, government, think tanks and the research community on September 18. The discussion was held under Chatham House Rule.

The 61 percent headline figure was generally unsurprising to the roundtable, and some participants said they had expected an even higher number. On how government, industry and researchers can meet each other's needs to act on the findings, there was less consensus.

Evaluation, access and biology

Survey results showed that evaluation of AI models is the mitigation experts trust most, ranking above technical safeguards, operational governance and international coordination. Participants shared that view and focused on what would make evaluation work.

Access emerged as a central barrier to independent evaluation. Some participants urged model developers to give evaluators access to their weights and full chain of thought, overriding privacy concerns. Others said sharing chain of thought with institutions such as the military carries its own security and privacy risks, and questioned what a reasonable request to companies would be. Some suggested a trusted-access arrangement.

Participants also called for more funding and capacity for evaluation science and standards, and for broad support for red teaming rather than capability evaluations alone. Proposals included:

• mandatory capability-based evaluation
• authorization and funding for the Center for AI Standards and Innovation (CAISI)
• a self-regulatory organization to set benchmarks for government use of AI

Biology drew particular concern. According to one expert, most AI safety evaluations today only assess large language models, while biological models lack proper evaluations. The greatest risk was seen in a biological model looped into a harness. Participants also flagged open-weight models as a growing area of importance and noted that nothing currently stops someone from running an experiment that combines AI and biology.

Different clocks and a workforce gap

Participants said the military, government and AI developers operate on different timelines. The military has historically focused on imminent rather than existential threats, frontier AI developers are racing toward future capabilities, and the government is playing catch-up. Several urged grounding the debate in what is deployed today and focusing on policies that can be implemented now. One participant argued that regulation should not slow innovation so much that small AI companies cannot win DoD contracts.

In the survey, 32 percent of respondents named workforce expertise and training among the most significant gaps in preparing for AI integration and defending against AI threats. The roundtable pointed to gaps in technical expertise in the military and at the State Department, and to a need for more education among policy leaders. Some saw hiring delays and bureaucracy resource cuts as the core problem, while others cited a lack of desire to enter government and the difficulty of returning after private sector work.

What it means

Experts converged on three areas for action: a dramatic increase in investment in AI evaluation science, especially beyond in silico testing; research led by scientists, not only technologists, on how AI risks affect the physical world; and a technologically literate workforce in the domains of greatest national security risk.

"The point of getting everyone in a room was to move from measuring the problem to figuring out how to solve it," IST's Philip Reiner said. IST says the findings are meant for the national security community, researchers, U.S. policymakers and frontier AI companies, and that the areas of consensus and disagreement show where momentum exists and where open questions remain.