Keepalived Dual-Master & VIP Failover Failures: Troubleshooting VRRP and Health Checks

Three Real-World Incidents

The article opens with three representative production incidents:

Incident A – VIP fails to failover: lb1's HAProxy crashes and restarts repeatedly, but VIP 10.x.x.100 stays on lb1 for 40 minutes. Root cause: health check script lost execute bit after a backup restore (tar without -p), so Keepalived silently skipped it at startup; additionally, a firewall hardening rule dropped outbound VRRP advertisements from lb1, so even if the check failed, priority changes never reached lb2.

Incident B – Split-brain (dual-master): Both nodes show MASTER for the same VIP, ARP flapping occurs. A third "ghost" node (10.x.x.13) cloned the production keepalived.conf without changing virtual_router_id or router_id, joined the same VRRP group with priority 150, causing constant master/backup oscillation.

Incident C – VIP moves but traffic fails: After a planned failover, logs show Entering MASTER STATE and ip addr confirms VIP moved, but cross-subnet clients time out for ~3 minutes. Upstream switch's ARP entry didn't update due to port-security rate-limiting GARP packets.

Terminology: Two Meanings of "Dual-Master"

Meaning 1 – Active-Active architecture (correct): Each node holds a different VIP (lb1 holds 10.x.x.100, lb2 holds 10.x.x.101) and backs up the other. Both VIPs are active simultaneously.

Meaning 2 – Split-brain fault (incorrect): Both nodes claim MASTER for the same VIP. Detection: ip -o -4 addr show | awk '{print $2, $4}' shows the same VIP on both nodes, or tcpdump -i eth0 -n -e vrrp shows advertisements from two source IPs.

Scope & Assumptions

Targets two-node Keepalived + VIP for L4/L7 load balancer HA (nginx, HAProxy, LVS) in self-managed data centers or cloud VMs where you control L2. Managed cloud SLBs are out of scope. Assumes Keepalived 2.2.x (RHEL9/Ubuntu 22.04), with notes for 2.3.x and 1.3.x differences. OS: Rocky Linux 9 / Ubuntu 22.04. Firewall: firewalld or nftables. Nodes in same L2 broadcast domain, VIP in same subnet.

Why Troubleshooting Goes Wrong

Four common mistakes:

Only checking if keepalived process is alive – most failures are logical (no advertisements received, health check always passes, wrong priority, stuck in BACKUP).

Tweaking config (weight, advert_int, fall) without first examining state machine logs.

Investigating only one node – VRRP is a two-party protocol; must compare both sides' states, logs, priorities, and packet capture.

Confusing "VIP moved" with "traffic switched" – Keepalived only moves VIP and sends GARP; upstream ARP, conntrack, and service readiness are separate.

Correct order: State machine → VRRP packets → Health checks → Upstream network .

Core VRRP Mechanics

6.1 VRRP is a Voting Protocol

Four key rules:

Priority (0-255) decides MASTER. 255 reserved for address owner; 0 means graceful relinquish (BACKUP takes over immediately, no Master_Down_Timer wait); 1-254 normal. If effective priority ≤ 0 → FAULT state.

MASTER sends advertisements every advert_int (default 1s); BACKUP only listens. Master_Down_Timer = 3 * advert_int + skew_time, where skew_time = (256 - priority) / 256. Lower priority → longer wait → higher priority fails over first.

Preempt is default. BACKUP receiving lower-priority advertisement immediately becomes MASTER – main cause of flapping. nopreempt prevents this.

New MASTER sends Gratuitous ARP (GARP). Failure here causes Incident C.

6.2 Packet Layer

VRRP uses IP protocol 112 (not a port), multicast 224.0.0.18, MAC 01:00:5e:00:00:12, TTL 255. Implications:

Firewalls must allow protocol 112, not "port 112".

TTL 255 means VRRP cannot cross L3 devices – must be same L2.

Multicast often dropped in cloud/VXLAN; use unicast_peer instead.

6.3 State Machine

States: INIT → (interface ready) → BACKUP ↔ MASTER, or → FAULT (track_interface down or track_script makes priority ≤ 0). FAULT means instance withdraws from election entirely; if both nodes FAULT, VIP disappears completely.

6.4 Two Independent Health Check Systems

vrrp_script (VRRP layer): Changes priority or forces FAULT → triggers VIP failover. Checks "can this node serve?" (nginx/HAProxy process, local socket).

LVS real_server checkers (MISC_CHECK, TCP_CHECK, HTTP_GET): Only add/remove backend servers from IPVS table. VIP does NOT failover on these failures.

Misassignment causes: backend failure triggers VIP flapping, or local process death doesn't trigger failover (Incident A).

6.5 Four Tracking Mechanisms

Configuration options and their effects: track_interface – Target: NIC. On failure: Instance → FAULT. Typical use: Business/uplink interface. vrrp_script + weight – Target: Custom script. On failure: Priority ± weight; if result ≤ 0 → FAULT. Typical use: nginx/HAProxy process. vrrp_script + weight 0 – Target: Custom script. On failure: Instance → FAULT (no priority change). Typical use: All-or-nothing scenarios. vrrp_sync_group – Target: Group of instances. On failure: Whole group fails over together. Typical use: Active-Active: keep VIPs together. weight sign matters: negative = subtract on failure (common); positive = add on success. Reversing sign is a frequent silent bug.

6.6 state and nopreempt

state MASTER/BACKUP

is only initial state; real role determined by priority and preempt. nopreempt only works with state BACKUP. Combining nopreempt with state MASTER yields undefined behavior across versions – must test with keepalived -n -l -D.

6.7 vrrp_strict and Firewall Interaction

vrrp_strict

enforces RFC compliance but inserts netfilter rules that conflict with firewalld reloads (rules get flushed) and hardening that blocks protocol 112. Fix: define dedicated nftables table/chain ( vrrp_nftables keepalived_vrrp) or explicitly allow VRRP in firewall permanently.

Four-Layer Diagnostic Framework

Layer 1: Process & Config – Pass: Instance in MASTER/BACKUP, config test passes. Failure: Stuck in INIT/FAULT; systemd restart loop.

Layer 2: VRRP Packets – Pass: Bidirectional advertisements seen, stable. Failure: No packets; unidirectional; rapid state flapping.

Layer 3: Health Checks – Pass: Manual script exit codes match Keepalived judgment. Failure: Script fails but priority unchanged; logs show failure but state static.

Layer 4: Upstream Network – Pass: Upstream device can ping/curl VIP. Failure: Local curl works, cross-node fails; stale ARP.

Production-Ready Dual-Master Config (Keepalived 2.2.x)

Topology: lb1 (10.x.x.11) holds VIP .100 primary, backs up .101; lb2 (10.x.x.12) vice versa. Both run HAProxy. Uses vrrp_sync_group to keep VIP pairs together. Both instances use state BACKUP + nopreempt so startup order doesn't matter. Priorities 150/100 with script weights -60 (or weight 0 for FAULT-on-failure) ensure single script failure triggers failover. GARP settings: garp_master_delay 1, garp_master_refresh 5.

Health Check Scripts – Hardened Patterns

Exit codes: 0=healthy, 1=unhealthy, 2=indeterminate (don't change state). Use set -uo pipefail (not -e), timeout on every external call, one-line log output. Example check_haproxy.sh checks process, stats socket (via socat), then HTTP endpoint fallback. Permissions: root:root 0755; directory chain must not be world-writable (Keepalived 2.x enable_script_security). Verify with journalctl -u keepalived | grep -i script.

Observability

Logs: Enable -D via systemd drop-in. Key grep patterns: Entering (MASTER|BACKUP|FAULT) STATE, effective priority, VRRP_Script, VRRP_Packet|checksum|TTL. Align timestamps across nodes with journalctl --since/--until -o short-iso | sort.

Metrics (Prometheus/node_exporter textfile): Prefer result-oriented keepalived_vip_held{vip="..."} 1|0 (shows actual IP on interface) over state metrics. Collect via notify script (event-driven) or periodic exporter script. Alert rules for split-brain ( sum by (vip) (keepalived_vip_held) > 1), VIP missing, flapping, stuck backup, FAULT state.

Ground truth commands: ip -o -4 addr show, ip neigh show, ip link show – never trust logs alone.

Five Closed-Loop Case Studies

VIP no failover (silent health check disable): Script lost exec bit after tar restore; Keepalived logged "Permission denied" once at startup then removed script. Fix: chmod 0755 + systemctl restart keepalived (reload doesn't re-load scripts). Also discovered firewall dropping protocol 112 on lb2 – fixed with firewalld rich rules for VRRP from peer IPs only.

Split-brain from ghost node: tcpdump showed third source IP (10.x.x.13) with same VRID/priority. Isolated rogue node at switch port. Prevention: VRID/router_id registry in CMDB, network segmentation, alert on unexpected advertisement sources.

Config change not applied (reload vs restart): Changed weight -30 to -60, ran reload, but failover still used -30. vrrp_script weight only applied at startup. Fix: restart (with capacity check on peer). Rule: changes to vrrp_script, virtual_router_id, interface, track_* require restart.

VIP moves but traffic blackhole: Upstream switch ARP entry stale due to port-security limiting GARP. Mitigation: garp_master_delay 0, garp_master_repeat 5, garp_master_refresh 10; long-term: static ARP on gateway or adjust switch ARP aging/port-security. Validate with cross-subnet curl loop and blackbox_exporter 5s probes.

Flapping from health check overload: check_haproxy.sh used socat to query HAProxy stats socket; under load, response time exceeded script's timeout 2, causing random failures. With fall 2 and 2s interval, oscillation period ~4-8s. Fix: raise timeout to 5s, fall 4; rewrite script to use lightweight checks (process, /proc/net/tcp, single TCP connect) and cache stats socket results. Add script duration metric.

Production Checklists

Active-Active requires each node sized for 100% traffic, connections must tolerate IP change (or use LVS connection sync), and vrrp_sync_group mandatory. virtual_router_id unique per L2 domain; router_id stable string (not IP). Scan for conflicts with tcpdump vrrp before deploy.

VRRP auth (PASS) is cleartext digest – prevents accidental mis-config, not attacks. Store password in 0600 file, not in repo.

Email notifications need local MTA; prefer notify script + Prometheus alerts.

High concurrency: consider net.ipv4.ip_nonlocal_bind=1, arp_ignore=1, arp_announce=2; graceful drain via notify_backup/fault background script with timeout.

All changes in change window, gray-release: standby node first, observe 24h, then active node, then immediate failover drill.

Backup config + scripts + state snapshot before every change; rollback = restore + restart (triggers failover). Emergency: manual ip addr add if Keepalived dead.

Verification Checklist (Post-Deploy/Change)

Version matches assumption ( keepalived --version).

Config test passes.

Interface name matches ip -br addr.

Scripts 0755 root:root, three exit codes verified.

Bidirectional VRRP seen ( tcpdump -i eth0 -n -e vrrp), only expected sources, correct VRIDs, TTL 255.

VIP distribution matches design (one VIP per node).

No flapping in 30 min ( grep -c 'Entering MASTER STATE').

Failure injection: stop HAProxy → priority drops, VIP moves, cross-subnet probe recovers in ≤4s.

Metrics exposed ( keepalived_vip_held), Prometheus rules loaded, test alert fires.

Quarterly drills: process kill, keepalived stop, interface down, config rollback – record failover latency.

Summary Rules

Troubleshoot in order: state machine → VRRP packets → health checks → upstream network.

VRRP has no port – protocol 112. Hardening checklists miss it; failure is silent.

VRRP needs same L2, TTL 255, multicast reachable; else use unicast_peer.

Priority and weight arithmetic must be validated together; weight 0 (FAULT on failure) avoids math errors. vrrp_script (VIP failover) vs real_server (backend pool) – never confuse. state is initial only; nopreempt requires state BACKUP – verify per version.

Reload does NOT apply vrrp_script, virtual_router_id, interface, track_* changes – restart required (triggers failover).

VRID uniqueness registry prevents ghost-node split-brain.

VIP moved ≠ traffic switched – verify from cross-subnet client; tune GARP refresh.

Health checks become failure source under load – monitor script duration, prefer kernel-level checks.

Active-Active = 100% capacity per node + sync group.

Every change: backup → syntax check → rollback plan → gray deploy → real failover drill. Silent failures demand metrics, drills, and process rigor.