Impact of High Network Latency on MySQL Group Replication Performance

Problem

MySQL Group Replication (MGR) uses a Paxos‑like protocol where consensus is reached as long as a majority of nodes agree. The question is whether, in a three‑node MGR cluster, a node with high network latency (e.g., located on another continent) can be isolated without degrading overall performance.

Experiment

A three‑node MGR cluster was set up (details omitted). Flow‑control was disabled on all nodes to allow unrestricted data traffic. A sysbench workload was run using the ts command to timestamp each output line. Network latency was then artificially added to the third node (mgr‑3), later removed, while performance metrics were recorded.

During the period when latency was introduced, the sysbench performance report showed a noticeable drop in overall throughput, which recovered after the latency was removed.

Analysis

The results contradict the intuition that a single slow node would not affect the cluster when flow‑control is off. The underlying reason lies in MGR’s use of the multi‑paxos protocol, which rotates the leader role among nodes. When a high‑latency node becomes the leader, other nodes must wait for its responses, creating a performance bottleneck.

Key points of the protocol:

MGR adopts multi‑paxos as its consensus mechanism.

Traditional Paxos has a single “dealer” node that initiates negotiations; in multi‑master mode, non‑dealer nodes forward transactions to the dealer, making the dealer a bottleneck.

Multi‑paxos rotates the dealer role, allowing each node to initiate negotiations independently, eliminating a fixed bottleneck.

However, if a node with high latency becomes the dealer, all other nodes must wait for it, degrading overall performance.

For a deeper dive, see the Oracle slide deck: Deep Dive into MySQL Group Replication .

Tips

When designing MGR clusters, consider the network latency of each node, as the rotating leader can cause latency‑induced performance drops.