Inside Google’s Production: How Requests Travel Through Its Massive Infrastructure

21CTO introduction: This is a Google SRE engineer’s May 2018 talk, outlining Google’s website infrastructure and code release process; for details see the book “Site Reliability Engineering”.

Network Layer

Google Edge network is a worldwide collection of servers (Data Centers) connected via the international backbone B4.

Inside a Data Center, the Jupiter network virtualizes hundreds of switches into a single switch with about 1.3 PB/s bandwidth.

A web request to Google first reaches the edge network, then the Google Front End (GFE), which acts as a reverse proxy and forwards the request to an internal service.

The Global Software Load Balancer (GSLB) performs load balancing on three levels: geographic location (e.g., google.com), product‑specific traffic (e.g., Maps, YouTube), and RPC calls.

The overall request path is: user → ISP → edge network → GFE → GSLB → Jupiter → target service, with latency measured in milliseconds.

Data Center

Google’s data‑center hierarchy is:

A campus can contain multiple data centers (similar to a region or availability zone).

Each data center consists of multiple clusters, shown as squares in the diagram.

A cluster contains many rows.

Each row holds many racks.

Each rack houses many machines.

Campus > Data center > Cluster > Row > Rack > Machine

Cluster Management

Google uses a job‑task model. A job defines the desired resources and consists of many subtasks. Engineers submit jobs via internal software, which forwards them to Borg, Google’s internal container‑orchestration system. The Borg master asks a scheduler which machines should run the job; the chosen machines (Borglets) then execute the tasks, automatically retrying on failure.

Tasks are located using the Borg Name Service (BNS) with paths like

/bns/<cluster>/<user>/<job name>/<task number>

. BNS mappings to IP:port are stored in Chubby, a distributed lock service that also provides a simple file‑system API and achieves consensus via Paxos.

Lock Service

Chubby stores the BNS‑to‑IP mapping and offers distributed locking for coordination across Google’s services.

Storage System

HDD + SSD: the physical storage layer.

D (Disk): manages HDD/SSD and provides temporary storage for running jobs.

Colossus: Google’s distributed file system, successor to GFS, offering permanent, replicated, encrypted storage.

Bigtable: a NoSQL database that stores ordered data and provides eventual consistency across clusters.

Spanner: a NewSQL database that combines relational semantics with NoSQL‑style scalability.

Monitoring

Borgmon is the monitoring system for Borg tasks. It aggregates task health information from each cluster, forwards summaries to a global Borgmon, a time‑series database, and an alert manager. When a cluster’s error rate spikes, alerts are triggered.

Prober periodically sends requests to task‑hosting servers and measures response times, feeding its data to Borgmon for an additional health perspective.

Inter‑task Communication

Tasks communicate via Stubby, an RPC service built on HTTP and protobuf. In practice, protobuf is used for RPC between tasks.

From Code to Production

Google’s monolithic repository is managed by Piper (similar to Git). By 2016 Piper handled 1 billion files, 2 billion lines of code, 9 million source files, ~86 TB of data, and 35 million commits.

When engineers submit code, they create a changelist and undergo review with Rietveld. After approval, a presubmit check runs static analysis and other tests before the code enters the repository.

Blaze (the internal build tool, open‑sourced as Bazel) compiles the code into binaries, requiring engineers to specify build outputs and dependencies.

Continuous testing pulls the latest code from the repo and runs automated tests. Successful builds are packaged by Rapid into an MPM (Midas Package Manager) package, which includes a name, version, and signature for authenticity.

Sisyphus, Google’s deployment tool, receives the MPM package and performs the production rollout, supporting strategies such as immediate, canary, or scheduled deployments.

Summary

Piper handles code submission; Blaze performs the build.

Rapid creates the MPM package.

Borg (using Chubby) runs the jobs.

User requests travel through edge, GFE, B4, GSLB, and Jupiter to reach a server.

Tasks communicate via protobuf and Stubby.

Spanner stores persistent data.

Borgmon and Prober monitor server health.

Future feature releases use Continuous testing, Rapid, and Sisyphus for deployment.