Sullivan & Cromwell builds Agreement Analyzer for contract review with Codex

From proof of concept to multi-user tool

Sullivan & Cromwell partner John Savva developed Agreement Analyzer as a single-user proof of concept built with Codex. He then worked with OpenAI to improve the quality of outputs, make the system more efficient through parallel processing, and turn the application into a tool that several lawyers can use at the same time.

The tool is meant to reflect how lawyers at the firm already work. Reviewing a draft agreement involves weighing its terms against relevant precedents, using approaches based on the firm's experience and keeping a client's priorities in mind. Agreement Analyzer brings those inputs together. It helps lawyers compare corresponding provisions across precedents and track how those provisions interact.

Savva says the use of AI at the firm has shifted as models have improved. AI was first used mostly for repetitive, administrative tasks such as drafting email responses or checking calendars. "Now, we're seeing many more opportunities for use in challenging legal tasks such as deep analysis of complex agreements," he says.

How the review workflow works

Agreement Analyzer integrates the firm's workflows, curated firm precedents, negotiating playbooks and ChatGPT Enterprise. A lawyer first inputs a draft agreement. The lawyer can then add selected precedents and a playbook that sets out firm and client positions, along with information on market practice.

The tool runs multiple review cycles from different perspectives, synthesizes the results and helps identify potential issues. It returns a memorandum and proposed tracked changes, which a lawyer can review, adapt or reject.

Savva stresses that the tool is not meant to replace lawyers. In his words: "It's not about replacing lawyers. It's about providing a tool for lawyers to do deeper analyses and find more authority and precedents to provide an even higher quality product for their clients."

Showing sources and uncertainty

Savva says transparency is what makes an AI tool's output trustworthy. "If an AI tool makes a claim about market practice, you need to be able to see the source of that information," he says.

He describes an application in which agents develop conclusions and other agents challenge them in an adversarial process. In one potential use case, specialized agents review the agreement in parallel for things such as market deviations and risk allocation, interactions between clauses, and ambiguities that could affect litigation.

The results separate three categories: findings supported after challenge by other agents, dismissed findings, and unverified issues that are retained with qualifications. Lawyers can inspect the issues, the explanations behind them and the relevant precedents, then decide which findings justify changes in the final markup.

What's next for the firm

Sullivan & Cromwell says it is continuing to explore tools shaped around specific legal tasks. For Savva, stronger analytical capabilities open the way to supporting more of the research and writing process.

He points to advances in reasoning with GPT-6 Astra: "Astra's really moved things forward in terms of the quality of the reasoning that's available, and that has been the bottleneck in the past," he says.

Savva expects AI to let lawyers "focus on the areas where their judgment is most important", so they can concentrate on the questions that matter most to their clients. The source does not give details on availability beyond the firm's own use, or on performance measurements for the tool.