How Facebook Migrated MySQL 5.6 to 8.0: Challenges, Patch Management, and Automation

Background and Motivation

Facebook announced that its long‑running upgrade to MySQL 5.6 took over a year, during which it continued developing the LSM‑tree storage engine MyRocks on the 5.6 branch. To avoid slowing MyRocks while moving to MySQL 5.7, Facebook chose to remain on 5.6 until MyRocks was complete, then planned a direct jump to MySQL 8.0.

Key Upgrade Challenges

Porting custom features to the new major version.

Ensuring replication compatibility between major releases.

Minimizing required changes to existing application queries.

Fixing performance regressions that hindered workload support.

Patch Management Strategy

Facebook created an 8.0 branch for building and testing, then began porting more than 1,700 patches from the 5.6 branch. Each patch was classified into one of four categories:

Drop: Features no longer needed or already present in 8.0, requiring no port.

Build/Client: Non‑server features such as modified MySQL tools (e.g., mysqlbinlog) or new client APIs that must be ported.

Non‑MyRocks Server: Server‑side features unrelated to the MyRocks engine that need migration.

MyRocks Server: Features that support the MyRocks storage engine and must be ported.

A spreadsheet tracked each patch’s status, history, and rationale for dropping. Grouped patches with similar functionality were ported together, and the commit messages in the 8.0 branch retained the original 5.6 messages for traceability.

Migration Path for Replica Sets

Facebook combined multiple mysqld instances into MySQL replica sets, each with a primary and several read‑only replicas distributed across data centers. The migration from a 5.6‑based replica set to an 8.0‑based one followed these steps:

Create logical backups with mysqldump and add them as 8.0 slave instances (no read traffic).

Enable read traffic on the 8.0 slaves.

Promote an 8.0 slave to primary.

Disable read traffic on the 5.6 primary.

Remove all 5.6 instances.

Each replica set could be migrated independently, with the ability to pause at any step and roll back if issues arose. Automation allowed configuration changes to move many replica sets through these stages with minimal manual effort.

Adoption of Row‑Based Replication (RBR)

As part of the 8.0 upgrade, Facebook standardized on row‑based replication because several 8.0 features require it and it simplifies MyRocks migration. Most replica sets already used RBR; the remaining statement‑based sets were converted after adding primary keys where needed.

Automation Verification

Facebook’s extensive automation framework ran integration tests on both 5.6 and 8.0 servers, exposing issues such as:

Parsing failures for error logs, mysqldump output, or SHOW command text.

Collation mismatches due to the default utf8mb4_0900 collation in 8.0.

Changed error codes for replication failures.

Removal of table.frm files in the new data dictionary, breaking legacy detection tools.

Need to support dynamic privileges introduced in 8.0.

Application‑Level Validation

To keep migration transparent to applications, Facebook built a shadow‑testing framework that replayed production traffic on 8.0 test instances. This uncovered:

Reserved‑keyword conflicts (e.g., GROUPS, RANK) causing parse errors.

Incompatible REGEXP behavior between versions.

Deadlocks on INSERT … ON DUPLICATE KEY statements after a 5.6 bug was fixed in 8.0.

Incompatibilities in custom document‑store and JSON functions, requiring compatibility shims.

ACL cache contention under high concurrent connections.

Binlog index contention during heavy write bursts.

Unexpected errors or timeouts on temporary‑table queries.

Increased memory usage due to the default performance_schema settings and the need to load InnoDB alongside MyRocks.

Memory usage was reduced by disabling unnecessary tools and tweaking InnoDB structures.

Future Work and Ongoing Challenges

The migration is still in progress, with many InnoDB replica sets already running on 8.0 and the remainder at various stages. Remaining hurdles include:

Logical dump and restore for very large mysqld instances, which can take days and require robust backup/recovery pipelines.

Detecting API changes that generate deprecation warnings in 5.7‑compatible clients.

Supporting dual‑version replica sets during transition, which complicates replication and feature compatibility.

Despite these obstacles, Facebook has observed benefits such as document storage, improved date‑time handling, and the ability to explore advanced MyRocks features like instant DDL.