How GitHub Upgraded Its 1200‑Node MySQL Cluster to 8.0 Without Downtime
GitHub detailed its year‑long, multi‑team effort to seamlessly upgrade over 1,200 MySQL servers—supporting more than 300 TB of data and 5.5 million queries per second—from 5.7 to 8.0, outlining the infrastructure, tools, and step‑by‑step migration strategy used to maintain service reliability.
GitHub recently shared its experience of seamlessly upgrading the underlying MySQL database of GitHub.com to MySQL 8.0.
GitHub stores massive relational data in MySQL; upgrading a fleet of more than 1,200 MySQL hosts without affecting the site’s Service Level Objectives was a complex task that required a year‑long planning, testing, and cross‑team collaboration.
GitHub’s MySQL infrastructure overview
Over 1,200 hosts, including Azure virtual machines and bare‑metal servers.
More than 300 TB of data, handling 5.5 million queries per second across 50+ database clusters.
Each cluster is configured for high availability with primary‑replica setups.
Data is sharded both horizontally and vertically; large‑domain workloads use Vitess clusters for horizontal scaling beyond single‑master MySQL limits.
A rich tooling ecosystem comprising Percona Toolkit, gh‑ost, Orchestrator, Freno, and internal automation for host‑cluster operations.
Because both MySQL 5.7 and 8.0 needed to run concurrently, GitHub’s tools and automation had to support mixed‑version environments and handle new, changed, or deprecated syntax between the versions.
To meet availability requirements, GitHub adopted a phased upgrade strategy with checkpoints and rollback capabilities. The plan consisted of the following steps:
Step 1 – Upgrade rolling replicas
Step 2 – Upgrade replication topology
Step 3 – Promote MySQL 8.0 hosts to primary clusters
Step 4 – Upgrade internal instance types
Step 5 – Cleanup
After confirming that clusters no longer need rollback and have successfully upgraded to MySQL 8.0, the remaining MySQL 5.7 servers are decommissioned. Validation runs for at least a full 24‑hour traffic cycle to ensure no issues during peak load.
The upgrade was driven by the impending end‑of‑life of MySQL 5.7 and the desire to obtain newer security patches, bug fixes, and performance improvements. GitHub also aims to leverage MySQL 8.0 features such as instant DDL, invisible indexes, and compressed binary logs.
For full technical details, see the original GitHub blog post: https://github.blog/2023-12-07-upgrading-github-com-to-mysql-8-0/
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
Programmer DD
A tinkering programmer and author of "Spring Cloud Microservices in Action"
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.