How to Prevent Split‑Brain in HA Clusters: Arbitration, Fencing, and Practical Strategies

Introduction

Split‑brain occurs when two nodes of a high‑availability (HA) system lose communication and each tries to claim shared resources, leading to chaos and possible data corruption. While stateless services can tolerate it, stateful services such as MySQL must strictly avoid split‑brain.

1. How to Prevent Split‑Brain?

Two common techniques are used:

Arbitration : When nodes disagree, a third‑party arbiter (e.g., a lock service, shared storage, or another system) decides which node should stay active.

Fencing : If a node’s state is uncertain, the fence mechanism forcibly disables it to release shared resources. Reliable fence hardware is required for this approach.

Ideally both arbitration and fencing are employed. In environments without shared resources (e.g., master‑slave database replication), fencing can sometimes be omitted, leaving arbitration as the sole safeguard.

2. Is It Safe Without a Fence Device?

Using PostgreSQL or MySQL replication as examples, the article examines client‑routing strategies:

VIP : Requires removing the virtual IP from a failed node; otherwise the node may continue to serve traffic and cause conflicts.

Proxy : Centralizes traffic through a single proxy, simplifying routing but demanding high availability for the proxy itself.

DNS : Unreliable because client caches may retain stale records.

Client‑side address list : Clients decide the primary based on server status; if two primaries appear, the original master should shut down its services.

When no physical fence device exists (common in cloud VMs), a soft fence such as an SSH‑based lock can be used to ensure a failed node releases resources after detecting loss of connectivity.

3. Can Data Be Lost After a Failover?

Data‑loss risk is separate from split‑brain. For PostgreSQL, synchronous streaming replication guarantees no data loss regardless of routing errors, because writes block until the replica acknowledges.

For MySQL, semi‑synchronous replication can be hardened by setting a very large rpl_semi_sync_master_timeout and keeping rpl_semi_sync_master_wait_no_slave enabled. If the replica fails, the master may hang; the solution mirrors PostgreSQL’s approach or uses a 1‑master‑2‑slave topology to avoid total data loss.

With asynchronous replication, some data loss is expected. In such cases, limit automatic failovers and prevent the former master from automatically re‑joining the cluster to avoid repeated data inconsistencies.

4. How to Implement the Above Strategies

You can script the logic yourself, but using mature cluster software is recommended. The article advises against Keepalived for stateful services and favors Pacemaker + Corosync with appropriate resource agents.

Key implementation points:

Understand the capabilities of resource agents (e.g., PostgreSQL agents support both synchronous and asynchronous replication and can auto‑switch between them; MySQL agents are weaker).

Ensure a quorum (majority) of nodes—typically at least three—to avoid split‑brain. Set no-quorum-policy=stop and avoid the unsafe ignore setting.

For two‑node clusters, add a third “arbiter” node or group several small clusters into a larger one with location constraints.

Alternatively, deploy a lock service (e.g., Zookeeper) as a pre‑emptive resource; the node that acquires the lock provides the service. The lock service itself must be highly available, often by running three instances and requiring a majority of locks.

Example lock‑service implementation (short‑connection HTTP): http://my.oschina.net/hanhanztj/blog/515065. A more robust version would use long‑living heartbeat connections to detect failures promptly.