How Uber Scaled 4,000 MySQL Instances with Docker: The Schemadock Architecture

Background

Uber's MySQL fleet is massive, exceeding 1,000 clusters and more than 4,000 database servers.

Problem

Initially managed with Puppet scripts and manual steps, the approach worked for a small number of clusters but became untenable at scale, prompting the need for a new management solution.

Key requirements for MySQL cluster management:

Ability to run multiple databases on each host.

Full automation.

A unified entry point to manage all clusters across data centers.

Solution

The team decided to build a Docker‑based system called Schemadock .

MySQL runs inside Docker containers, each representing a node. Cluster topology is defined in configuration files.

For example, a cluster named Cluster A is described in a config that lists its databases, their roles, and which instance is the master.

Agents on each host read these specifications and create/configure the containers accordingly.

Schemadock also includes a centralized service for overall maintenance and monitoring of instance health and drift.

Why Docker?

Containerization makes it easy to run multiple databases—potentially different versions and configurations—on a single host, saving resources.

With MySQL inside containers, the host’s role is simplified to “run containers,” eliminating Puppet‑specific dependencies.

Docker also simplifies testing; the entire cluster‑setup process can be exercised in a test environment.

Uber notes that Docker is best suited for large‑scale deployments; for smaller setups (fewer than ~16 MySQL clusters) tools like Puppet or Ansible remain sufficient.

Stateless MySQL Images

The MySQL image is deliberately stateless: it contains no replication or health‑check logic. Container role is supplied via environment variables, separating logic from the image.

Data resides on host‑mounted directories, not inside the container.

Benefits of this approach include:

If a container fails, it can be discarded and a new one launched with the same parameters.

Upgrading MySQL is as simple as swapping to a newer image.

Configuration changes are easy to apply.

Container Orchestration and Topology Configuration

Each MySQL container is started with a role and placed in the correct position of the replication topology as defined by the config.

An agent process runs on every host, receives the target specification, and creates/configures containers accordingly.

Example specification:

"schemadock01-mezzanine-mezzanine-us1-cluster8-db4": {
  "app_id": "mezzanine-mezzanine-us1-cluster8-db4",
  "state": "started",
  "data": {
    "semi_sync_repl_enabled": false,
    "name": "mezzanine-us1-cluster8-db4",
    "master_host": "schemadock30",
    "master_port": 7335,
    "disabled": false,
    "role": "minion",
    "port": 7335,
    "size": "all"
  }
}

This means host schemadock01 will run a Mezzanine MySQL instance as a slave on port 7335, with its master at schemadock30:7335 .

The agents run in an infinite loop, waking every 30 seconds to verify that containers match their specifications.

Check if a container is running; if not, create and configure it.

Ensure containers are in the correct replication topology; if a slave needs to become master, verify the old master is read‑only and GTID is fully applied before promotion.

Validate role‑specific parameters (e.g., master must be writable).

Start or stop auxiliary containers such as heartbeat or deadlock monitors.

Agents do not enforce any ordering when configuring containers; they act on whichever container they encounter first.

If a slave is processed before its master is ready, the agent pauses configuration for that slave and revisits it in the next loop once the master becomes available.

Illustrative diagram:

Summary

The Uber MySQL Docker solution consists of four main components:

Stateless MySQL containers.

Configuration files describing the overall cluster topology.

Agents on each host that create, configure, and continuously verify containers.

A centralized service for maintenance and monitoring.

This architecture achieves the original goals: multiple databases per host, full automation, and a unified management entry point.

Migration to Docker began in early 2016; today Uber runs about 1,500 Docker servers hosting roughly 2,300 MySQL databases.

Content curated from Uber's official blog: https://eng.uber.com/dockerizing-mysql