Mastering High Availability Clusters: Key Concepts, Resource Management, and Failure Handling

HA (High Availability) clusters provide redundancy for front‑end directors, back‑end RS‑servers, database servers and shared storage by using active‑primary and passive‑standby nodes, so that if a primary fails the backup takes over immediately.

Primary (active) and backup (passive) nodes are defined for the director, RS‑servers and storage such as MySQL, each with its own failover resources.

HA focuses on reliability and stability; availability is calculated as service‑up time divided by (service‑up time + downtime) and expressed in nines (99%, 99.9%, …) with five‑nines required for some financial systems.

When a node fails, its resources (VIP, services, devices, filesystems) must be transferred to another node. Resource stickiness (preference) and constraints determine which node receives the resources.

Stickiness scores >0 indicate preference; negative scores force migration. Constraints include colocation (whether resources can share a node) and location (node‑specific scores). The higher combined score wins.

Order constraints define start/stop sequences for dependent resources such as VIP before IPVS rules.

Resource types:

Primitive – runs on a single node.

Clone – runs on every node.

Group – moves as a unit.

Master/Slave – runs on two nodes, one primary.

Backup nodes detect primary failures via heartbeat messages. In clusters with three or more nodes, a quorum voting mechanism decides node legality; odd numbers of votes are preferred, and weighted votes can be assigned.

When a node is deemed illegal, actions include Freeze (process existing requests only), Stop (halt services and migrate resources), or Ignore (continue running, used only for two‑node pairs).

For a MySQL service, required resources are VIP, floating IP, MySQL service, and a mounted filesystem.

Split‑brain scenarios occur when multiple nodes write to the same file after an unsynchronized failover; preventing this requires node isolation (e.g., power‑off via STONITH) and storage isolation (e.g., FC‑SAN).

Node failure detection methods include shared‑disk arbitration, gateway ping, and watchdog timers.

The messaging layer (UDP/694) transports heartbeats, stickiness, and constraints; the Cluster Resource Manager (CRM) decides resource placement and orchestrates migrations via its components PE (policy engine), TE (transaction engine), and LRM (local resource manager). Resource agents (RA) implement start/stop/status scripts (LSB, OCF, etc.).

Typical HA stack combinations are:

haresource + heartbeat v1/v2

crm + heartbeat v2

pacemaker + corosync

pacemaker + heartbeat v3

cman + ragmanager

A minimal web‑service HA cluster needs at least two nodes running the messaging layer and CRM, and defines four resources: VIP, HTTP service, filesystem, and STONITH device.

Configuration tips: node names must match uname -n, use /etc/hosts for name resolution, synchronize time, enable SSH trust, and ensure CRM services start automatically while application services are managed by CRM.