OpenAI built Pando, a Git backend that cut median push latency nearly 5x

Why OpenAI replaced its source control backend

OpenAI says the size and demands on its monorepo grew at a "staggering pace" over the past year. The company links this to Codex, which lets engineers do far more work in parallel. That means more branches, pushes and merges, plus more fetches and clones from development environments, research jobs and CI.

According to OpenAI, these demands compound: one engineer can now have several changes ready for CI at once, and each push can set several jobs fetching the same commit. Starting around the end of last year, the systems supporting the monorepo struggled to keep up. OpenAI reports slow fetches and pushes along with failed merges, with reliability at times falling below 90%.

The team asked whether it had hit a limit of Git itself or of the way the monorepo was served. It concluded the problem lay in the backend, and chose to keep the familiar Git workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers at once.

How Pando works

OpenAI describes two goals for Pando: horizontally scalable writes, optimized for concurrent pushes to many branches, and maximum read throughput, so clients can download data fast enough to fully saturate their available network bandwidth. Pando remains compatible with the Git protocol and client, so both engineers and agents could adopt it within existing workflows. OpenAI says the migration happened without interrupting engineering work.

Two design choices stand out:

• Per-branch coordination. Pando coordinates updates to each branch independently, so updates to unrelated branches can proceed in parallel. The tradeoff, OpenAI notes, is that a reader looking at several branches can see them at different points in time.
• Packfile reuse. Pando reuses packfiles, Git's format for compressing multiple objects together, across fetches instead of generating an individual packfile for each client. A client may download some objects it already has, but the server can serve the same bytes to many clients with little computation.

Together, OpenAI says, these choices let it scale writes across branches and serve reads from caches.

Alternatives OpenAI evaluated

The company modeled two common approaches to scaling Git before building Pando. The first coordinates ref updates across full repository replicas using three-phase commit (3PC). A majority of replicas must accept each ref update, which requires multiple rounds of communication. OpenAI says this means references are updated one at a time, and that waiting for a quorum puts replica disk performance and communication latency on the push path.

The second uses a shared write-ahead log (WAL) manifest in blob storage to order updates across the repository. The source text available here ends partway through the description of this design, so its tradeoffs are not covered.

OpenAI also notes that systems imposing a single order on all writes to a repository can make pushes to unrelated branches wait on one another, which is the problem Pando's per-branch approach is meant to avoid.

What it means

For OpenAI, the result is a source control system sized for agent-scale work while engineers keep using standard Git. The company says staying with Git meant client improvements could be useful beyond OpenAI, and that it wants to contribute improvements from its copy of Git to upstream. It also says it will discuss what comes next and how it will keep contributing to Git to support agent-scale work.