How OpenAI Scaled Habitat Storage on Azure Cosmos DB for 1 Billion Users

Habitat at OpenAI's scale

According to OpenAI, all of its products depend on fast, reliable data access provided by Habitat, its online storage platform. The company says Habitat has to respond in milliseconds to over 1 billion users across 40 geographic regions, and that it has grown from a simple client library and database into a globally distributed platform serving over 500 petabytes of data.

Part I of the series covered Habitat's evolution from a Python client to a service. Part II focuses on the underlying storage component, Azure Cosmos DB. OpenAI says that through a close working partnership with the Azure team it arrived at a viable multi-tenancy approach of co-managed containers, with excellent read performance at massive, global scale.

OpenAI describes its priorities as a sequence: first become reliable enough for mission-critical product traffic, then fast enough for global users, and finally operate at massive scale.

Phase I: One shared container and its limits

OpenAI explains that single-tenancy, where each product team runs its own system, limits blast radius and gives clear ownership, but becomes expensive and often leads to significant over-provisioning. Multi-tenancy puts a central database team in control, but brings noisy-neighbor problems, where one tenant takes an unusually large share of resources.

At first, Habitat used a single default Azure Cosmos DB container in each account and let any client load through, relying on the stated 10K RU/s limit per logical partition key to provide back pressure. OpenAI lists three problems with this setup:

• Logical partition key limits: Because early API choices exposed partition key selection to clients, different product use cases often collided on common keys such as user_id or workspace_id. They then shared limits and constrained throughput for a given user.
• Noisy neighbors: Limits are actually enforced at the physical partition, so logical partitions cannot reach 10,000 RU/s in practice and share capacity with colocated keys. OpenAI says workloads for niche product features disrupted core login and signup flows.
• Weak partitions: A container's provisioned throughput is spread roughly evenly across its physical partitions. When partitions split as data grows, total throughput stays the same, so the limit per partition shrinks. For storage-heavy, blob-like use cases, storage growth alone reduced throughput for active workloads.

The company also notes that before moving to Habitat and Azure Cosmos DB, scaling Postgres suffered from hot organizations or keys that could take out the whole database, which required significant effort on caching and leases for cache misses.

Phase II: Isolated containers and their costs

To address these issues, OpenAI began using separate containers for the most critical new use cases. Each container gets its own physical partitions, so only workloads on that container can be noisy neighbors. Combined with a script that synchronizes each container's provisioned throughput with its number of physical partitions, this mostly avoided the problems above, according to the company.

Many legacy critical use cases, however, remained stuck in the default container without a data migration. As pressure and production incidents grew on that container, OpenAI began provisioning containers for any new service or large product feature. This, it says, saved the company from many site-wide outages and limited the blast radius to feature or service degradation.

The overhead eventually caught up. Each container needs individual capacity planning. In the shared container, a new use case going from 0 QPS to 1 million QPS was unlikely to shift total usage much. On an isolated container, the same jump might represent 1,000x more load than the baseline from internal dogfooding. Teams had to pre-provision for launches, but often forgot, and OpenAI says it sat in SEV rooms on launch day trying to scale containers at the last minute.

What it means

OpenAI says Habitat is today a multi-tenant database layer for many of its products, and that it reverted to a multi-tenant container by default. The company adds that it is still very early in its multi-tenancy story and has deferred investment in this area by using Azure Cosmos DB's built-in primitives wherever possible.

For teams building large shared data platforms, the account illustrates the tradeoff between isolation and efficient capacity planning. The source text is cut off before the final design details of the co-managed container approach are described.