OpenAI's Pando Cuts Git Push Latency 80% at Petabyte Scale

Problem

Before Pando, OpenAI's monorepo suffered from slow fetches and pushes, frequent merge failures, and reliability sometimes dropping below 90%. Two bottlenecks were identified: many branches pushed concurrently forced unrelated updates into a single queue, and many readers fetching the same history caused the server to compute a fresh pack for each reader.

Architecture

Pando decouples writes from reads. Unrelated branches update independently; a push to one branch does not wait for another. Readers can view branches at different points in time.

Write path

Pushed packs flow as immutable files into object storage. The server records only the new branch tip, the pack, and its dependencies. Metadata write volume does not grow with push size.

Read path

On fetch, the server matches the client's haves and wants against version records, selects the needed packs, and returns a packfile-uri for direct download. The response body is an empty pack. In one hour, the system served 57 TB, with over 99% of bytes coming from local disk cache.

Pack compaction

A background process merges small packs into larger ones. The monorepo currently retains four compaction layers, each kept to serve clients at different synchronization points.

Pack reuse via main-branch broadcasting

By default only the main branch is broadcast to clients. This makes client-produced packs depend only on the main branch, so later clients that already have that main-branch segment can directly download the same pack. Continuous pushes to the same branch, or stacking feature branches on other feature branches, require retransmitting objects outside the main branch.

Implementation and rollout

The system is written in Rust. From project start to production traffic took 10 weeks (traffic began April 2026). Core design was done by one engineer, implementation by AI agents, deployment by a small team. OpenAI is exploring sharing the technology but no installable version exists yet. Client-side changes to Git's packfile-uri have been partially upstreamed.

Results

Median push latency reduced to nearly 1/5 of the original (≈80% reduction).

Pack sizes: median < 20 KiB, 99th percentile < 3 MiB.

Daily Git data transfer approaches 1 PB.

Takeaway for other teams

If your agents push many branches concurrently, observe whether the server still locks the entire repository for each push. If it does, decouple branch updates and broadcast only the main branch so that packs produced by one push can be reused by subsequent fetches.

Source: https://openai.com/index/scaling-git-at-openai/

Code example

👀点击关注
AI前沿快速Follow