Facebook’s MySQL 5.6‑to‑8.0 Migration: Challenges, Process, and Lessons Learned

Facebook reports that their most recent major upgrade to MySQL 5.6 took more than a year and that they continued developing the MyRocks LSM storage engine on the 5.6 branch.

Because building a new storage engine for MySQL 5.7 would have slowed MyRocks progress, they chose to stay on 5.6 until MyRocks was finished and then upgrade directly to MySQL 8.0.

The official blog notes that this migration was more challenging than previous upgrades.

MySQL, an open‑source database developed by Oracle, powers many of Facebook’s critical workloads. Facebook contributes new features to client connectors, storage engines, the optimizer, and replication. Each new major MySQL version requires substantial effort, including porting custom functionality, ensuring replication compatibility, minimizing query changes, and fixing performance regressions.

The last major upgrade was to MySQL 5.6, which took over a year to release. While MySQL 5.7 was available, Facebook kept developing MyRocks on 5.6. As MySQL 8.0 was released, they began finalizing the migration of MyRocks to the UDB service layer.

MySQL 8.0 introduces notable features such as write‑set parallel replication and an atomic DDL‑supporting transactional data dictionary. Migrating to 8.0 also brings missed 5.7 features like document store.

1. Code Patches

Facebook created an 8.0 branch for building and testing, then began porting the more than 1,700 patches from the 5.6 branch. The patches were organized into four categories:

Drop – features no longer needed or already present in 8.0.

Build/Client – non‑server features that support the build environment, such as modified MySQL tools or new asynchronous client APIs.

Non‑MyRocks Server – server‑side features unrelated to the MyRocks engine.

MyRocks Server – features that support the MyRocks storage engine.

Patch status and history were tracked in spreadsheets, and related patches were grouped for porting. About 1,500 of the 2,300 evaluated patches were successfully ported to 8.0.

2. Migration Path

Multiple mysqld instances were combined into a MySQL replica set, each with a primary and replicas distributed across data centers. The migration steps are:

Create a logical backup with mysqldump and add it as an 8.0 replica (no read traffic).

Enable read traffic on the 8.0 replica.

Promote the 8.0 replica to primary.

Disable read traffic on the 5.6 primary.

Remove all 5.6 instances.

Each replica set can be migrated independently, and groups of replica sets are migrated in stages, allowing rollback if needed.

3. Row‑Based Replication

As part of the 8.0 migration, Facebook standardized on row‑based replication (RBR) because several 8.0 features require it and it simplifies MyRocks porting. Replica sets still using statement‑based replication (SBR) were gradually converted to RBR.

4. Automated Verification

Most of the migration was validated using Facebook’s automation framework, which runs integration tests (canaries) against both 5.6 and 8.0 servers. A dedicated test environment with VM‑based replica sets was built to verify behavior. Issues discovered included:

utf8mb4_0900 collation mismatches causing replication and schema‑validation problems.

Changed error codes that required automation updates.

Removal of table.frm files in the 8.0 data dictionary, breaking some tools.

New dynamic privileges in 8.0 that needed automation support.

5. Application Validation

Facebook built a MySQL shadow‑testing framework that captures production traffic and replays it on test instances. This uncovered problems such as reserved‑keyword conflicts, REGEXP incompatibilities, repeatable‑read deadlocks on INSERT … ON DUPLICATE KEY statements, and incompatibilities in custom document‑store and JSON functions.

Additional operational issues included ACL cache contention, binlog index contention under heavy write loads, temporary‑table query failures, and increased memory usage due to the default performance_schema settings and InnoDB structures. Memory usage was reduced by disabling unnecessary tools and modifying InnoDB internals.

6. Ongoing Work

Most custom features are now on 8.0, but migration is still in progress. Skipping the 5.7 major version introduces challenges such as detecting API changes, handling large logical dump/restore processes, and managing replica sets that run dual versions. New 8.0 features like the utf8mb4_0900 collation can break replication with 5.6 instances.

Despite the obstacles, Facebook has observed benefits from running MySQL 8.0, including access to document store, improved date‑time support, and the potential to add instant DDL capabilities to MyRocks.