How YouTube Scales Billions of Videos with Vitess and Distributed Databases

Introduction

YouTube is the second‑largest website after Google, with over 500 hours of video uploaded each minute in May 2019 and more than a billion hours of video watched daily.

The platform serves over 2 billion users, generating billions of page views each day.

Backend Infrastructure

YouTube’s backend microservices are written in Python, Java (using Guice), Go, and JavaScript for the UI.

The primary database is MySQL clustered with Vitess for horizontal scaling, complemented by Memcache for caching and Zookeeper for node coordination.

Popular videos are delivered via a CDN, while less‑frequent videos are fetched directly from the database.

Each uploaded video receives a unique identifier and undergoes a batch job that generates thumbnails, metadata, transcodes, and monetization settings.

Advanced codecs such as VP9 and H.264/AVC reduce bandwidth requirements for HD and 4K streams.

Dynamic Adaptive Streaming over HTTP adapts video quality to the viewer’s connection speed, minimizing buffering.

Why Vitess Was Needed

Initially YouTube used a single MySQL instance, but growing QPS required horizontal scaling.

Master‑slave replication added read replicas to reduce load on the master, but replicas could serve stale data.

To handle further growth, sharding was introduced, distributing data across multiple machines and increasing write throughput.

Disaster management was added to replicate data across geographically distributed data centers for redundancy and low latency.

Vitess: Horizontal Scaling for MySQL

Vitess is a database clustering system that runs on top of MySQL, providing built-in sharding, automatic failover, backup, and query rewriting for performance.

Vitess uses a Go‑based connection pool to manage MySQL connections efficiently and relies on Zookeeper for cluster state.

Deploying to the Cloud

Vitess is cloud‑native and runs well in containerized environments, being Kubernetes‑aware.

YouTube runs Vitess on Kubernetes, leveraging Google Cloud Platform services such as Cloud Spanner, Cloud SQL, Cloud Datastore, and Memorystore.

Content Delivery Network (CDN)

YouTube uses Google’s global network of edge POPs to deliver content with low latency and cost.

Data Storage

Videos are stored on disks managed by Google File System (GFS) and BigTable, while metadata and relational data reside in MySQL.

Commodity Servers and Storage Disks

Google data centers use homogeneous, off‑the‑shelf servers that are inexpensive to replace and scale.

Primary storage is rotating hard drives for cost‑effective petabyte‑scale capacity, while SSDs are used for performance‑critical workloads.

Key hardware metrics include high I/O speed, security compliance, large capacity, acceptable cost, and reliable low‑latency operation.