USC engineer warns AI is not ready for critical infrastructure
AI systems are not ready to run critical infrastructure, according to Neil Siegel, a USC Viterbi engineering professor whose career includes work on ambulance dispatch networks, air traffic control and the Minuteman missile system. He says the systems’ unpredictable errors create a dangerous gap between consumer software practices and the demands of infrastructure where failures can cost lives.
- Neil Siegel says AI errors cannot be reliably bounded in frequency or magnitude.
- He warns that third parties may already be deploying AI in critical infrastructure.
- Siegel distinguishes consumer software practices from mission-critical engineering.
- He calls for an FDA- or FAA-style regulator with testing and failure thresholds.
A reliability gap between consumer software and critical systems
Siegel made the comments in an interview filmed for The BlackVeil Files, with the full interview available on YouTube. He pointed to his experience with the Minuteman missile system, which involved 450 nuclear warheads over 60 years with zero accidental launches.
His central concern is that the mathematics behind current AI engines cannot establish how often errors will occur or how severe they will be. “The algorithms used in these AI engines are based on mathematics that cannot, absolutely cannot, bound either the frequency or the magnitude of the errors they make,” Siegel said.
He described a system that can produce reasonable answers most of the time but still fail unpredictably. “You ask 10 questions, and nine times it makes a pretty reasonable answer. And the tenth is wildly wrong,” he said. “And if you depend on the tenth, you would die.”
Why deployment by others worries Siegel
Siegel distinguishes Silicon Valley’s “toy business,” which he associates with speed-to-market and features, from mission-critical engineering. A temporary social media outage may be an annoyance, he said, while an ambulance dispatch outage can cost lives.
He also contrasted frequent software updates with the controls applied to aircraft. “They send out a patch to their software every 17 seconds,” Siegel said. “In our business, we don’t do that. Because if you patch an airplane, you have to redo the airworthiness certificate. That costs $100 million.”
His concern is not necessarily that the companies building AI engines will place them directly in ambulance dispatch systems. “My worry isn’t that the people who build these AI engines will put them in an ambulance dispatch system. My worry is that somebody else will. And they probably already are.”
A call for regulation and explicit uncertainty
On artificial general intelligence, Siegel said he teaches why he does not think it is possible, citing Kurt Gödel’s incompleteness theorems, Alan Turing’s proof concerning problems that no algorithm can solve, and Roger Penrose’s theory about the human brain operating at a non-deterministic quantum scale.
He argues that AI systems are financially encouraged to guess rather than disclose uncertainty. In his view, critical infrastructure should instead require systems to avoid guessing and clearly signal when they are unsure.
Siegel calls for a regulatory body modeled on the FDA or FAA, with specifications, failure thresholds and testing protocols. “You can’t even sensibly talk about an AI infrastructure without it,” he said. For users and operators, his warning is that impressive performance on ordinary tasks does not establish the reliability required for systems on which lives depend.
USC Viterbi (AI): When Failure Is Not an Option: Neil Siegel on AI’s Dangerous Blind Spot
