2 Engineers, AI, and a Rust Rewrite: Scaling OpenAI's Storage to 1B Users

What Is Habitat

Habitat is OpenAI's unified online storage platform. It started in mid‑2024 as a simple Python client library connecting ChatGPT's main service to Azure Cosmos DB. Its goal: let product engineers store and retrieve data without worrying about data‑type handling, routing, auth, encryption, serialization, request shaping, or connection‑pool management.

Because it was easy to adopt, Habitat spread rapidly across product teams even without a mandate. Two years later it has grown into a distributed system that handles:

70,000,000+ requests per second

1 billion+ weekly active users

500 PB+ of data

Coverage across nearly 40 geographic regions

OpenAI's growth rate exceeds 10× every year, leaving the Habitat team almost no breathing room for "future‑proof" architecture — every step is "fixing the gun while fighting the battle."

From Library to Service: A Textbook Operational Incident

By mid‑2025 the client‑library model had hit a wall. As logic grew more complex and the number of internal services exploded, even a backward‑compatible protocol upgrade became nearly impossible. The article recounts a real incident that illustrates the core problem:

Add new routing logic in the client behind a feature flag.

Coordinate rollout of the new client to dozens of services — took days.

Run shadow‑traffic validation of the sharding logic — more days.

Discover a bug, fix, re‑release — more days.

When finally flipping the feature flag, one team rolled back to an old client version with a known bug, directly causing the regional outage the migration aimed to prevent.

This story shows the fundamental flaw of the client‑library model: any change requires fragile coordination across dozens of services, and failure probability scales linearly with team count.

OpenAI therefore decided to extract Habitat into a standalone service.

Three Key Benefits of Service Extraction

Unified release and observability entry point: Instead of updating dozens of clients, change one service and all products benefit immediately.

Centralized security and compliance control: Access policies, audit logs, and limits on underlying storage (e.g., Cosmos DB) are enforced in one layer, regardless of whether the caller is external, internal, or an AI agent.

Future‑proof platform evolution: No longer blocked by a lagging client holding back the overall upgrade cadence.

Why Python Despite Known Limitations

The team knew Python would increase network latency and CPU/memory scaling costs compared to an in‑process library. They explicitly acknowledged that Python's inefficiency would be unacceptable at 100× scale, making a future rewrite inevitable. Yet they chose Python first as a strategic technical debt — the immediate priority was unblocking product teams and stabilizing the platform foundation. As the original post states:

We placed a "calculated bet": betting that our internal code models would advance rapidly enough to dramatically simplify the future migration path — by the time we actually needed to migrate, Codex and GPT should be capable of doing the job.

That bet paid off.

Four Battles of Running a High‑Concurrency Python Service

When a single user request triggers hundreds of database calls, the slowest call becomes the user‑visible latency. The article details four real‑world incidents.

Battle 1: asyncio Scheduling Latency — Concurrency ≠ Parallelism

asyncio handles I/O efficiently but cannot bypass the GIL, so it offers no true CPU parallelism. Habitat also performs CPU‑heavy work: routing, compression, encryption, checksums, downstream health checks, shadow traffic, and request hedging.

During tail‑latency investigation, traces showed requests often "stuck" waiting for a coroutine to be rescheduled to parse the response, even though Cosmos DB had already replied quickly.

Conclusion: Under high CPU load, asyncio event‑loop scheduling jitter can reach hundreds of milliseconds, sometimes seconds. The team learned to monitor event‑loop busyness explicitly and keep per‑process concurrency very low, scaling horizontally with many Python worker processes instead.

Battle 2: A Feature‑Flag Config Causing "On‑the‑Hour Stalls"

Online CPU profiling revealed another tail‑latency culprit: periodic parsing of Statsig (feature‑flag / A/B testing tool) config files. Two defaults combined to create a storm:

Statsig pulls the full production rule set for all services every minute with no jitter.

Each pod runs up to 8 Python processes to improve CPU utilization and latency.

Result: every minute, all workers in a pod stall simultaneously parsing the huge config, leaving in‑flight requests waiting.

Fix: deliver smaller, targeted configs; increase refresh interval; add random jitter to background tasks.

Battle 3: Connection‑Pool LIFO Triggering "Metastable Failure"

To keep scheduling latency low, requests must be evenly distributed across service processes. Poorly tuned client connection pools break this balance: a high‑throughput client may open only a few connections, directing all its traffic to a handful of service processes.

Before fixing load‑balancing, the team saw utilization variance where some "tail" processes handled 5–10× the average concurrent requests.

A real incident: even after stopping the traffic‑spiking client, some service processes remained in a degraded "metastable" state, worsening until manually restarted — a known failure mode called metastable failure .

Root cause: Python aiohttp 's TCPConnector defaults to LIFO (last‑in‑first‑out) connection reuse. Recently returned connections are preferred, which normally helps recycle burst‑created connections quickly. But here it created a vicious feedback loop:

Under load, a few connections become "hot" and are reused repeatedly.

The processes behind those connections get overloaded, increasing latency.

Higher latency causes clients to hold connections longer, reinforcing the hot‑connection pattern.

Even after the traffic source disappears, the hot connections stay hot, keeping the associated processes overloaded.

The team first validated the hypothesis by limiting max connection reuse time, which mitigated the degradation. Then they switched the pool strategy from LIFO to FIFO (first‑in‑first‑out) , breaking the loop and also reducing steady‑state request variance.

Today OpenAI relies on Istio and Envoy for connection pooling and load‑balancing that understands actual server‑side load, eliminating this class of problem at the infrastructure layer.

Battle 4: Downstream Connection Storms and Envoy Connection Aggregation

The many Python processes spawned to reduce asyncio latency created a side effect: they easily overwhelm downstream dependencies with massive connection counts — the classic thundering‑herd problem.

A poorly timed routine release could cause significant CPU jitter from bulk connection rebuilds; a connection leak could saturate NAT gateways and take down the network. Because Habitat's process count is an order of magnitude higher than typical services, resource‑planning thresholds based on steady‑state throughput were far too low.

Solution: Envoy . Upgrade Python‑side HTTP/1 connections to HTTP/2, leverage multiplexing, and centralize connection pooling and lifecycle management. Envoy also becomes the single place to enforce rate‑limiting and circuit‑breaking — far more effective than scattering those policies across thousands of Python processes.

"Habitat Deliberately Does Less": Constrained API for Predictable Performance

Another reason Python scaled this far is that Habitat's API is intentionally limited, making per‑request overhead predictable.

Habitat does not expose arbitrary SQL; it provides a simple NoSQL API. This is a deliberate trade‑off : a system with simple, predictable, constant‑cost requests is far easier to scale and harder to misuse than a powerful but unpredictable one. Requests with unbounded fan‑out are the real danger — they complicate isolation, load‑balancing, and create latency cliffs that are hard to scale against.

The article recalls the pre‑Habitat era when most online data lived in Postgres. Small teams could manually review every query and schema change to ensure index usage and controlled behavior. As teams and products grew, that "human gatekeeping" collapsed; a single expensive query could bring down the entire database.

The core issue is cost asymmetry : writing a costly, hard‑to‑handle SQL query is too easy. Habitat makes "expensive" operations explicit at the client layer — no unbounded queries, complex joins or graph traversals are pushed to the product layer, forcing more efficient designs.

Concretely, Habitat exposes a NoSQL API built around client‑defined Object and Edge types , inspired by Meta's TAO paper. Clients pre‑define objects, edges, and relationship types (but not the content of each type), forming a graph‑like schema. However, Habitat does not support typical graph traversal queries — only direct edges of an object can be queried.

Data partitioning colocates an object with its edges in the same storage partition, but does not colocate the object with the remote objects it points to. This makes horizontal scaling natural; the trade‑off is lower graph‑traversal efficiency because each hop may cross different Cosmos DB accounts, possibly in different regions.

For clients that truly need complex queries, Habitat offers an "escape hatch": Change Data Capture (CDC) streams online mutations in near real‑time to a dedicated Rockset instance. Each client team scales its own Rockset cluster for analytical/search workloads. This adds some integration cost but is considered the right trade‑off: keep simple queries simple, give an exit for complex needs, and isolate read‑heavy analytical load from the online store.

Highlight: 2 Engineers, One Quarter, Full Rust Rewrite

Delaying the Python rewrite by a year let the team focus on more urgent, higher‑impact problems. But as the platform matured and growth accelerated — Habitat became OpenAI's second‑largest service by core count (Envoy deployment ranked fourth) — crossing the Python barrier became a priority.

At its peak, the Python version sustained over 20 million requests/second .

The climax arrived in Q2 2026: just 2 engineers, aided by Codex and GPT‑5.5, rewrote the entire service in Rust within one quarter.

The new Rust service now handles 95% of production traffic ; the Python version will be fully retired within weeks. OpenAI reports:

CPU efficiency improved 6×

Memory efficiency improved 15×

Both average and tail latency dropped significantly

Looking back at the four Python battles — asyncio scheduling, feature‑flag jitter, connection‑pool LIFO, downstream connection storms — it's clear why the numbers are so striking: Rust eliminates at least three of those root causes at the language level (no GIL, no asyncio scheduling latency, native efficient memory and connection management).

Achieving a core production service language migration with 2 people in one quarter validates the team's year‑old bet: when the time came, Codex and GPT were already strong enough.

Summary: Engineering Decision Philosophy Under Extreme Growth

This OpenAI engineering blog is worth rereading not for "new tech" but for its demonstration of decision‑making under extreme growth pressure:

Accept phased technical debt, but know exactly what you're betting on and when to pay it back: Use Python for speed now, bet that internal code models will make future migration feasible.

Trade constraints for scalability: Habitat deliberately avoids a "powerful" API, pushing complexity to clients explicitly, gaining predictability and system‑wide scalability.

Separate operational complexity from language efficiency: Solve the "library vs. service" architecture problem first, then the "Python vs. Rust" language problem — tackling both simultaneously only makes things harder.

AI coding capability now handles core infrastructure migrations: 2 engineers + Codex + GPT‑5.5 rewriting a service handling tens of millions of QPS in one quarter — an engineering scale barely imaginable a few years ago.

OpenAI teases a Part II covering multi‑tenant reliability, layered read‑performance optimization, and deep collaboration with Azure Cosmos DB to sustain this growth trajectory.

Original article: https://openai.com/index/scaling-storage-one-billion-users-part-one/