Why GitHub’s MySQL Cluster Crashed: Lessons from Recent Outages
GitHub experienced multiple service outages over recent weeks due to resource contention in its MySQL1 cluster, leading to prolonged downtimes, and the company disclosed detailed timelines, root causes, and ongoing mitigation measures such as load audits, traffic shifting, and infrastructure scaling to prevent future incidents.
In recent weeks GitHub suffered several service outages caused by resource contention in its MySQL1 cluster, which degraded many platform services.
GitHub announced detailed information about these incidents on March 23.
Timeline
Mar 16 14:09 UTC – outage lasting 5 hours 36 minutes
Mar 17 13:46 UTC – outage lasting 2 hours 28 minutes
Mar 22 15:53 UTC – outage lasting 2 hours 53 minutes
Mar 23 14:49 UTC – outage lasting 2 hours 51 minutes
The primary cause was contention for resources in the MySQL1 cluster during peak load, affecting many services and features.
Over the past years GitHub has added clusters, partitioned its primary database, and made other optimizations, but these measures are not a permanent fix, and the issue is still being actively addressed.
To prevent future incidents, GitHub is auditing the database’s load patterns during peak periods, shifting traffic to other databases, accelerating failover times, and reviewing change‑management procedures, especially monitoring and changes during high‑load production periods.
GitHub will continue to expand its infrastructure, including sharding the database and scaling hardware, as the platform grows.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
MaGe Linux Operations
Founded in 2009, MaGe Education is a top Chinese high‑end IT training brand. Its graduates earn 12K+ RMB salaries, and the school has trained tens of thousands of students. It offers high‑pay courses in Linux cloud operations, Python full‑stack, automation, data analysis, AI, and Go high‑concurrency architecture. Thanks to quality courses and a solid reputation, it has talent partnerships with numerous internet firms.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.