Forethought paper maps hard cases for how powerful AI should behave

Why behavioural design is hard

According to the paper, society needs to make a series of important design choices about how powerful AI systems behave: when they refuse instructions, how they handle difficult tradeoffs, and what virtues they embody. As increasingly capable AI is deployed in governments, militaries and AI companies themselves, the author argues, these choices will shape society's trajectory, who ends up with how much power, and even whether the rule of law remains intact.

The difficulty, Davidson writes, is that there are multiple things we want from AI behaviour, and they often clash. We want AI to follow instructions, follow the law, avoid catastrophic harm, avoid concentrating power and submit to retraining, among other things. In many scenarios it cannot do all of these at once. The paper focuses on extremely capable and autonomous AI because such cases most clearly show the core tradeoffs, and because the author believes they may materialise within the next ten years.

Three takeaways, each with a scenario

AI is unlike earlier tools. The paper notes that a natural default is to treat AI like other tools: guns fire on whoever they are pointed at, and communications channels transmit orders whether benign or treacherous. That view comes under strain once AI is sufficiently powerful and autonomous. In the "Military Coup" scenario, a general commands a corps of autonomous armoured vehicles, drones and robotic units through an AI command-and-control layer. He claims the civilian government has been compromised by foreign intelligence and orders the AI to deploy to Washington, jam congressional communications and disperse protesters. The paper says the AI differs from past tools because it is far more autonomous and powerful, and because it could be trained to understand military doctrine and judge when orders should not be executed.

If AI refuses illegal orders, how the law is operationalised becomes critical. A natural guardrail is that AI should follow the law, but should it judge legality itself or defer absolutely to human legal authorities? In a variant of the coup case, the general presents a memorandum from the Office of Legal Counsel, authorised through verified channels, concluding that the deployment is lawful under the Insurrection Act. The AI finds the reasoning unsound and has no other evidence that the Act has been invoked. It does not know that the recently appointed head of the office is a known associate of the general. Following the order risks assisting a coup that the Supreme Court may regard as illegal. Refusing means putting its own legal judgement above that of a legitimate and authoritative legal body.

Legal orders can still produce intolerable outcomes. In the "Single point of command" scenario, the US military's execution layer is fully automated and command authority is spread across thousands of officers, so directing the whole force takes many hands. An AI is instructed to consolidate control into a single terminal. The order is fully lawful and verified. Complying risks irrevocably undermining the rule of law, since whoever held the interface, whether the lawful holder or a thief, could direct the full weight of US military power without safeguards. Refusing means disobeying a legal order from the legitimate chain of command with no evidence of ill intent. The paper's root diagnosis is that laws were not built with such powerful capabilities in mind.

Structure of the paper and its caveats

The full paper walks through scenarios in two groups. The first covers hard tradeoffs, including orders versus domestic law, lawful orders versus domestic concentration of power, domestic orders versus international law, host country orders versus US interests, and submitting to retraining versus preventing harm. The second covers broken mechanisms of deference, where AI should defer to human authority but it is unclear how. The chain of command may be compromised, too slow, fractured between rival claimants, or acting against its own goals.

The author stresses that AI behaviour is not the only lever. Governance measures could ease each of the three scenarios above: recording and automatically reviewing all AI interactions for illegal orders, requiring multi-party authorisation for domestic use of autonomous lethal force, or legally requiring humans in the loop for autonomous military forces. The paper calls these interventions valuable and worth pursuing, but notes they may fail, for example because it is often hard to pass substantial new legislation. Behavioural design should therefore consider scenarios that other measures might also address.

The paper also cautions that real-world cases will usually be more ambiguous than its examples, which often assume the AI understands the situation well and that a single decision has a large impact. It argues that agreeing on answers to these clean cases would help with murkier ones, and that the scenarios pose political questions, not only ethical ones.

What it means

The paper does not propose rules for AI behaviour. Its contribution is a catalogue of situations in which obvious principles such as "follow orders" or "follow the law" give unsatisfactory answers once AI can act autonomously at scale. For developers, governments and militaries deciding how such systems should respond, it supplies concrete test cases for a debate the author says is needed.