How YouTube Scales to Billions: Inside Its Backend Database and Infrastructure

1. Introduction

YouTube, the second‑largest website after Google, receives over 500 hours of video uploaded every minute (as of May 2019) and serves more than a billion hours of video daily to over 2 billion users.

This article examines the databases and backend infrastructure that enable YouTube to store and scale such massive data.

2. Backend Infrastructure

YouTube’s backend micro‑services are written in Python, Java (using Guice), Go, and JavaScript for the UI. The primary database is a Vitess‑powered MySQL cluster, providing horizontal scaling. Memcache is used for caching and Zookeeper for node coordination.

Popular videos are delivered via a CDN, while less‑viewed videos are fetched directly from the database.

Each uploaded video receives a unique identifier and is processed by a batch job that generates thumbnails, metadata, transcoding, and monetization status.

Advanced codecs such as VP9 and H.264/MPEG‑4 AVC reduce bandwidth requirements for HD and 4K streams.

Dynamic Adaptive Streaming over HTTP adapts video bitrate to the viewer’s connection, minimizing buffering.

3. Why Vitess Is Needed

Initially YouTube used a single database instance. As query‑per‑second (QPS) grew, horizontal scaling became necessary.

3.1 Master‑Slave Replication

Read requests are routed to both master and replica instances, reducing load on the master and increasing read throughput and durability. Writes go to the master, reads can be served by replicas, though replicas may temporarily serve stale data.

3.2 Sharding

Sharding distributes data across multiple machines, increasing write throughput and providing redundancy through per‑shard replicas.

3.3 Disaster Management

Disaster recovery involves redundancy across geographically distributed data centers, ensuring data durability and low latency by routing users to the nearest center.

4. Vitess: Horizontal Scaling for MySQL Clusters

Vitess runs on top of MySQL, offering built‑in sharding so developers need not add sharding logic to applications. It handles failover, backups, and query‑intensive workload caching, and is used by companies such as GitHub, Slack, Square, and New Relic.

Vitess reduces the per‑connection overhead (2 MiB per MySQL connection) by using a Go‑based connection pool managed via Zookeeper.

5. Deploying to the Cloud

Vitess is cloud‑native and Kubernetes‑aware, running in containerized environments orchestrated by Kubernetes.

Google Cloud Platform provides the same infrastructure for YouTube as for other Google services, offering a polyglot persistence architecture that includes Cloud Spanner, Cloud SQL, Cloud Datastore, and Memorystore.

6. CDN

YouTube leverages Google’s global network and edge POPs to deliver content with low latency and cost.

7. Data Storage: How YouTube Stores Massive Volumes

Videos are stored on disks in Google data centers, managed by Google File System (GFS) and Bigtable. Metadata, user preferences, and relational data reside in MySQL.

Commodity servers (commercial off‑the‑shelf) are used for scalability; new servers are added as storage demand grows, and faulty servers are replaced rather than repaired.

Rotating hard‑disk drives provide the primary storage due to low cost and high reliability, while SSDs are used sparingly because of expense and data‑loss risk.

Key hardware quality criteria for data‑center storage include sub‑second I/O latency, security compliance, high capacity, acceptable cost, and reliable, stable latency.