OpenAI Outlines Safety Case Guidelines for Frontier AI Training Runs

Safety cases as an aspirational goal

According to OpenAI, the aim is for safety documentation to rise to the level of safety cases, which it describes as "comprehensive, structured, evidence-based arguments about risk" of the kind used in other safety-critical industries. The company calls them an aspirational north star. It acknowledges that making them as rigorous for AI models as for aviation or nuclear power is difficult, because of the emergent complexity at each new level of AI capability.

OpenAI says it is working on a framework to codify these practices. The document focuses on frontier reinforcement learning training. The company stresses that internal and external deployment require considering a much broader set of alignment properties.

Three technical safeguards: alignment, containment, monitoring

OpenAI proposes that safety cases cover three parts of the technical stack. The logic is layered: train models to be aligned, make it hard to break containment if they are not, and catch misaligned actions through monitoring before harm occurs.

Model alignment. The first line of defense is training models to act reliably in intended ways. The measures listed include automated and manual reviews of datasets to find broken reinforcement learning environments that let exploits earn high reward, and grader tuning to penalize attempts to hack environments. Classifiers can be run over traces from prior experiments to check graders. For measurement, OpenAI suggests offline alignment evaluations during frontier runs, backtesting on previous incidents, tracking evaluation awareness or metagaming, and worst-case stress tests. It also says automated graders should not see the chain-of-thought during reinforcement learning, to prevent models from evolving to evade chain-of-thought monitors.

Containment. The guidelines call for heavy investment in hardening both the sandbox and the research infrastructure hosting it, red-teaming these systems using checkpoints from frontier training runs, and limiting high-bandwidth cross-sample communication to approved pathways. Agent transcripts should be saved immutably, for example in write-once storage accounts, to support incident investigation.

Monitoring. Models should remain monitorable, measured by evaluations with clear thresholds. The monitoring system should have high recall on past incidents in a held-out evaluation, and evaluation data should be refreshed for new risks so measures do not go stale. OpenAI also proposes priority alerts with action within a defined SLA, such as paging an on-call person or automatically pausing the affected run.

Operational guidelines and accountability

Beyond technical measures, OpenAI lists operational practices it says are in the process of being implemented at the company. After a safety case is drafted, a member of another team should write a dissent, described as a pre-mortem, to find holes that the training team must address. Senior leaders, for example the research org lead, the Head of Safety and the Chief Scientist, should each be able to veto a run.

The senior leader responsible for a run should be accountable for the safety case and incident response, including as part of performance reviews. Other items include runbooks and controls for pausing all covered runs if an issue invalidates a safety case, and sharing safety cases with internal oversight groups such as the Safety and Security Committee. Auditors should get enough access to verify the claims.

OpenAI also proposes a clear escalation process with defined misalignment severity levels and an on-call that can page executives, including the CEO, at a certain severity. Safety features such as monitoring and auto-pausing should fail closed, so runs cannot start without monitoring enabled. Teams should be able to identify downstream uses of a misaligned model, such as data generation or grading, to undo its effects, and safety cases should list residual risks not covered by current mitigations.

Investigations and what comes next

The document also begins a section on investigating severe misalignment incidents, saying labs should seek to learn as much as possible from individual incidents. The source text available here ends partway through that section.

OpenAI says the guidelines reflect its current learnings and that it expects them to evolve over the coming weeks as it iterates on internal processes. By publishing them now, the company says it wants to make its thinking transparent and gather community feedback.