How to Prevent Split‑Brain in High‑Availability Clusters: Practical Strategies & Scripts

In actual production environments, you can prevent split‑brain problems by following several approaches:

Use both serial and Ethernet cables for connections and maintain two heartbeat lines so that if one fails the other continues to transmit heartbeat messages.

When a split‑brain is detected, forcibly shut down one heartbeat node (requires special hardware support such as Stonith or feyce), causing the standby node to lose the heartbeat and send a power‑off command to the primary node.

Implement robust monitoring and alerting (email, SMS, on‑call) so that administrators can intervene immediately; for example, Baidu’s alert system distinguishes upstream and downstream alerts, allowing admins to reply with simple commands that the server processes automatically.

When designing high‑availability solutions, assess whether your business can tolerate the potential loss; for typical websites the loss is usually acceptable.

In multi‑node clusters, add arbitration mechanisms to decide which node should acquire resources. Common ideas include:

Introduce an arbitration IP: when heartbeats are completely lost, each node pings the reference IP; the node that can reach it takes over the service.

Use third‑party arbitration software (similar solutions exist at Alibaba).

Enable a disk lock: the active node locks the shared disk, preventing the other node from stealing it during a split‑brain. Intelligent locking only activates when all heartbeat links are down.

Provide enough time for human operators to handle alerts before the server automatically takes over; for instance, trigger an alert within one minute but delay automatic takeover to five minutes, allowing manual intervention.

Below is a reference script for handling split‑brain situations using a third‑party arbitration mechanism based on pinging a gateway IP.

Key notes:

The script cannot resolve split‑brain caused by firewall or configuration issues; it only handles network‑related split‑brain and assumes the gateway server is stable.

VIP acquisition failures can be due to keepalived service faults (non‑split‑brain) or communication problems (split‑brain).

Script usage steps:

Check if the VIP is obtained; if not, verify keepalived status and restart if necessary.

Periodically ping the gateway (third‑party arbitration IP). If reachable, ensure keepalived is running; if not reachable for a threshold, stop keepalived.

Implementation details:

#!/bin/bash
export PATH=$PATH:/usr/sbin
# Split‑brain check and control: third‑party arbitration via pinging gateway IP
CHECK_TIME=3               # Number of attempts
VIP=$1                     # Virtual IP passed as argument
GATEWAY=192.168.1.8       # Gateway IP (adjust as needed)
eth=enp2s0                # Local network interface
keepalived_communication_status=1
get_vip_status=1
keepalived_service_status=1
active_status_str='active (running)'

echo "Starting script check_gateway.sh $VIP; time:"
date

function check_get_vip_status() {
  if [ $(ip add | grep "$VIP" | wc -l) -eq 1 ]; then
    get_vip_status=1
  else
    get_vip_status=0
  fi
  return $get_vip_status
}

function check_keepalived_status() {
  /sbin/arping -I $eth -c 5 -s $VIP $GATEWAY >/dev/null 2>&1
  if [ $? = 0 ]; then
    keepalived_communication_status=1
  else
    keepalived_communication_status=0
  fi
  return $keepalived_communication_status
}

function check_keepalived_service_status() {
  if [ $(systemctl status keepalived.service | grep "$active_status_str" | wc -l) -eq 1 ]; then
    keepalived_service_status=1
  else
    keepalived_service_status=0
  fi
  return $keepalived_service_status
}

while [ $CHECK_TIME -ne 0 ]; do
  check_get_vip_status
  if [ $get_vip_status = 0 ]; then
    CHECK_TIME=0
    check_keepalived_service_status
    if [ $keepalived_service_status = 0 ]; then
      echo "Running script check_gateway.sh $VIP; starting keepalived service"
      systemctl start keepalived.service
    fi
    echo "Running script check_gateway.sh $VIP; VIP not obtained, exiting. Time:"
    date
    exit 0
  fi
  let "CHECK_TIME -= 1"
  check_keepalived_status
  if [ $keepalived_communication_status = 1 ]; then
    CHECK_TIME=0
    check_keepalived_service_status
    if [ $keepalived_service_status = 0 ]; then
      echo "Running script check_gateway.sh $VIP; starting keepalived service"
      systemctl start keepalived.service
    fi
    echo "Running script check_gateway.sh $VIP; GATEWAY=$GATEWAY, communication OK, exiting. Time:"
    date
    exit 0
  fi
  if [ $keepalived_communication_status -eq 0 ] && [ $CHECK_TIME -eq 0 ]; then
    echo "Running script check_gateway.sh $VIP; stopping keepalived service"
    systemctl stop keepalived.service
    echo "Running script check_gateway.sh $VIP; GATEWAY=$GATEWAY, communication failed 3 times, stopped keepalived. Time:"
    date
    exit 1
  fi
  sleep 3
done

Schedule the script via cron (e.g., run every minute with a 10‑second delay after system boot):

* * * * * sleep 10; bash /usr/local/src/check_gateway.sh 192.168.1.198

Additional reference image:

(Copyright belongs to the original author; removed upon request)