Keepalived + Nginx HA: Deployment, Failover Verification, and Pitfall Avoidance

Background: HA Is More Than a VIP

Setting up Keepalived + Nginx is easy, but real-world failures reveal missing closed loops: stale backend configs on standby, connection drops during fail-back, dual-master due to network blips, heartbeat on business network causing split-brain, Nginx process hangs undetected, and untested notify scripts. The article organizes a closed-loop workflow: goal → environment → pre-checks → implementation → config explanation → activation → verification → risks → rollback → maintenance.

Environment & Version Constraints

OS: Rocky Linux 9 / AlmaLinux 9 (RHEL 9 compatible) — commands for RHEL 9; Debian differences noted

Keepalived: 2.2.x / 2.3.x / 2.4.x — syntax compatible with 2.2.4+; 2.4.3 adds auth_hmac Nginx: 1.30.x (stable) / 1.31.x (mainline) — from nginx.org official repo; keep versions identical

Nodes: 2 (primary/backup) — two nodes cannot achieve quorum during split-brain

Network: Business net + dedicated heartbeat net — heartbeat must be L2 reachable, not cross L3

VIP: Free address in business subnet — must verify unused

Firewall: firewalld / nftables / iptables — allow VRRP protocol (IP protocol 112)

Key version notes: Keepalived 2.x differs greatly from 1.x ( vrrp_sync_group, virtual_server syntax changes; default VRRPv2 for IPv4, VRRPv3 for IPv6). vrrp_strict forbids authentication blocks and forces no_accept. authentication ( auth_type PASS) is deprecated per RFC 3768; real security comes from auth_hmac (2.4.3+) or network isolation. nopreempt and state MASTER are mutually exclusive — using both causes nopreempt to be ignored with a warning.

Core Knowledge: VRRP Mechanics & Failover Vacuum

3.1 VRRP Basic Model

Master holds VIP, answers ARP, forwards traffic. Backup listens for advertisements. Virtual router identified by virtual_router_id; all nodes in same virtual router must share same VRID and VIP list. Work loop: Master sends VRRP advertisement every advert_int (default 1s). Backup resets master_down_interval timer on receipt. After 3 × advert_int + skew_time without advertisement, Backup declares Master down. Highest-priority Backup becomes Master, binds VIP, sends gratuitous ARP (GARP). Upstream switches/hosts update ARP caches, traffic shifts. skew_time = ((256 - priority) / 256) × advert_int, staggering Backup actions so higher priority acts first.

Key conclusions: Minimum failover detection time is 3 × advert_int (3s default). Reducing advert_int to 0.5s cuts detection to 1.5s but raises false-failover risk. Post-failover traffic recovery depends on ARP convergence; GARP may be dropped by switch port-security, ARP suppression, or ARP proxy, causing traffic to stick to old MAC until ARP timeout (~300s). Countermeasure: increase GARP repeat count and interval via

global_defs

vrrp_garp_master_repeat

, vrrp_garp_master_delay, vrrp_garp_master_refresh, vrrp_garp_master_refresh_repeat.

3.2 Why Failover Breaks Connections

VIP failover is transparent for new connections (after ARP convergence) but catastrophic for existing connections: TCP long-lived connections (DB, gRPC, WebSocket) see RST/timeout; HTTP keep-alive connections in client pools fail on next request; local session state (sticky sessions) lost on new Master. True HA means sacrificing existing connections to guarantee new ones. For connection-sensitive workloads: externalize sessions (Redis/DB), implement client-side retries with idempotency, add connection-pool health checks, use multiple VIPs with client-side load balancing.

3.3 Split-Brain: Inherent Two-Node Limitation

Split-brain = both nodes think they are Master, VIP bound on both. Causes: heartbeat link failure (cable, switch port, optics, NIC), firewall blocking VRRP (protocol 112, not TCP/UDP), switch multicast/broadcast suppression dropping heartbeats. Consequences: MAC flapping, random traffic distribution, data corruption if backend stateful. Two-node VRRP cannot solve split-brain at protocol level because it cannot distinguish peer death from link loss. Engineering mitigations: (1) dedicated heartbeat link (direct cable or isolated VLAN); (2) third-party arbitration (etcd/Redis registration); (3) auto-convergence via priority gap + preemption so higher-priority node reclaims VIP after link restore. Monitoring: on each node, alert if both hold VIP simultaneously ( ip -4 addr show dev eth0 | grep -c 'VIP').

Architecture Design & Pre-Flight Checks

4.1 Target Architecture

Client → VIP 10.0.0.100 → [nginx-01 (MASTER, prio 150) ←heartbeat 10.10.0.0/24→ nginx-02 (BACKUP, prio 100)] → upstream app cluster 10.0.1.11-13:8080

Key points: both Nginx configs must be identical (same template, only node-specific params differ); VIP and node IPs same subnet, gateway to upstream switch; heartbeat on separate subnet, preferably direct or isolated VLAN; backend health handled by Nginx max_fails, Keepalived only checks local Nginx serviceability.

4.2 Pre-Flight Checklist

VIP truly free: ping and arping ( arping more reliable as some devices block ICMP).

VRRP protocol allowed:

firewall-cmd --add-rich-rule='rule protocol value="vrrp" accept'

(protocol name, not port).

Nginx can start on Backup: if config uses listen VIP:80, Backup fails because VIP not bound. Solutions: (A) listen 0.0.0.0:80 (recommended) or (B) sysctl net.ipv4.ip_nonlocal_bind=1 (risk: multiple processes may bind same VIP:port silently).

ARP kernel params: net.ipv4.conf.all.arp_ignore=1, net.ipv4.conf.all.arp_announce=2 (mainly for LVS/DR but reduces ARP confusion).

Baseline consistency: time sync ( chronyc), Nginx/Keepalived versions, backend config diff (critical — config drift makes HA useless).

NTP required for

auth_hmac

anti_replay time

mode; clock skew must stay within time_window (default max(3× advert_int,5)s).

Implementation Steps

5.1 Installation

RHEL: dnf install keepalived; add nginx.org repo with module_hotfixes=true to bypass AppStream module filtering. Debian: apt install keepalived nginx; must purge default keepalived.conf placeholder to avoid duplicate instances.

5.2 Global Config

global_defs { router_id NGINX_01 script_user root enable_script_security vrrp_version 3 vrrp_garp_master_delay 2 vrrp_garp_master_repeat 3 vrrp_garp_master_refresh 30 vrrp_garp_master_refresh_repeat 2 vrrp_skip_check_adv_addr vrrp_strict }

enable_script_security

enforces script ownership/permissions (root, not group/other writable). vrrp_strict disables authentication blocks and forces no_accept; use auth_hmac instead.

5.3 Health Check Script

Design principles: check real business path (not just process), per-step timeouts, log failures, use fall / rise for hysteresis. Script checks: (1) nginx process exists; (2) local health endpoint /nginx-health returns 200; (3) upstream probe /api/ping — only logged as WARN, not fatal, because if upstream totally down, both nodes would go FAULT and VIP would be unheld, worse than serving 502s. Nginx health config includes access_log off to avoid 170k daily log lines. Script permissions: root:root, 755; log dir /var/log/keepalived 750; logrotate with copytruncate (script appends, doesn't reopen).

5.4 VRRP Instance Config

Primary (nginx-01, 10.0.0.11): state BACKUP, interface eth0, virtual_router_id 51, priority 150, advert_int 1, unicast_src_ip 10.0.0.11, unicast_peer {10.0.0.12}, auth_hmac with key file:/etc/keepalived/keys/vrrp.key (32-64 bytes), virtual_ipaddress {10.0.0.100/24 dev eth0}, track_interface {eth0}, track_script {chk_nginx}, nopreempt 1, notify_master/backup/fault scripts. Backup (nginx-02) differs only: priority 100, unicast_src_ip 10.0.0.12, unicast_peer {10.0.0.11}. Keys:

mkdir -p /etc/keepalived/keys; chmod 700; openssl rand -hex 32 > vrrp.key; chmod 400

. auth_hmac mode migration: receive-only → permissive → enforce for zero-downtime rollout.

5.5 Key Parameter Trade-offs

state MASTER

overrated: only sets initial state; real Master decided by priority+advertisements. Two MASTERs with different priorities still converge, but dual MASTER config risks dual-master during network anomalies. Recommended: both state BACKUP, priority differentiates. nopreempt prevents fail-back on recovery, avoiding double interruption and flapping. Requires state BACKUP; if state MASTER, nopreempt ignored with warning. Trade-off: Master may stay on weaker node; needs monitoring. advert_int: default 1s (3s detection). 0.5s gives 1.5s detection but doubles advertisement rate and false-failover risk, especially in virtualized/cloud environments. Keep 1s; optimize at application layer. weight vs priority: effective priority = priority + weight (negative on failure). Must ensure priority+weight < peer priority to trigger failover. Example: primary 150, backup 100, weight -40 → effective 110 > 100, no failover. Correct weight ≤ -(150-100+1) = -51; use -60 with margin. With fall 3, interval 2 → worst-case 6s detection+failover. track_interface: add business NIC (eth0) so NIC failure triggers immediate FAULT and failover. But if heartbeat shares same NIC (not recommended), NIC flap causes flapping. With dedicated heartbeat NIC (eth1), unicast_src_ip uses eth1 IP, track_interface only eth0. Single-NIC architecture: omit track_interface, rely on chk_nginx (local curl fails when NIC down).

5.6 Notify Script

Logs state changes, VIP presence, and on FAULT includes recent health-check failure lines. Must be fault-tolerant: all external calls with --max-time, wrap with timeout 5, never return non-zero (else Keepalived logs error but state machine unaffected).

Activation & Verification

6.1 Syntax Check

keepalived -t

(or timeout 5 keepalived -n -l -D for debug). Common errors: missing interface, typo unicast_src, vrrp_strict with authentication conflict.

6.2 Start & Observe

systemctl enable --now keepalived

; journalctl -u keepalived -f. Primary should show Transition to MASTER STATE; backup only Entering BACKUP STATE. If both show MASTER transition, stop one immediately — dual-master causes MAC flapping.

6.3 Status Checks

journalctl

for latest MASTER/BACKUP/FAULT; ip -4 addr show dev eth0 | grep VIP; tcpdump -i eth0 -nn 'ip proto 112' to verify advertisements (backup should only see peer's ads; seeing own ads indicates split-brain).

6.4 Four Verification Scenarios

Passive failover (node crash): poweroff primary; observe backup VIP appearance within 3-5s, MASTER transition, ping VIP, business request recovery <5s. If >10s, check advert_int /heartbeat latency; if VIP up but ping fails, check GARP suppression.

Active failover (Nginx failure): systemctl stop nginx on master; watch health-check log for three FAILs (interval 2s), then priority drop, Entering BACKUP STATE. Verify backup takes VIP.

Recovery & no-preempt: restart nginx on original master; with nopreempt + state BACKUP, it should stay BACKUP, not reclaim VIP. If it reclaims, check for nopreempt warning.

Heartbeat loss (split-brain drill): ip link set eth1 down or iptables -I INPUT -p vrrp -j DROP. Both become MASTER, VIP on both — expected. Restore heartbeat; higher priority wins. With nopreempt, split-brain won't auto-converge; must manually stop one. Alternative: preempt_delay 300 delays reclaim 5min, allowing auto-convergence while avoiding immediate fail-back.

Strategy comparison: default preempt (immediate fail-back, auto-heal split-brain), nopreempt (never fail-back, split-brain requires manual intervention), preempt_delay (delayed fail-back, auto-heal after delay). Recommended: preempt_delay.

6.5 Measured Failover Duration

Real measurement: continuous curl from client, trigger failover, compute outage window. Typical: health-check triggered (stop nginx) 6-10s (3×2s detection + failover + ARP); whole-node crash 3-5s (3×1s + ARP); business NIC down 2-4s. Health-check slower by design to avoid false positives.

6.6 Nginx Hang Scenario

Process alive but not accepting connections (worker freeze, socket exhaustion). pgrep passes but curl times out → health-check fails → failover after 3 failures. Verifying with kill -STOP on master PID simulates hang; kill -CONT restores.

Risks & Rollback

7.1 Change Risk Matrix

First Keepalived start: may seize VIP → IP conflict → arping verify free

Change priority: may trigger failover → seconds outage → maintenance window, keepalived -t first

Change virtual_router_id: instance leaves cluster → dual-master → change both simultaneously, verify single master

Change advert_int: sensitivity change → false failover or longer outage → adjust incrementally, observe 24h

Change auth_hmac key: auth failure → all go FAULT → sync change or three-phase migration

Stop Keepalived: VIP released → outage if peer not ready → confirm peer in BACKUP first

systemctl restart keepalived: VIP briefly released → unnecessary failover → prefer reload

7.2 Rollback Procedures

Config rollback: keep timestamped backups, cp back, keepalived -t && systemctl reload keepalived. Note: reload rebuilds VRRP instance, causing DELETED→BACKUP→MASTER transition with seconds of outage. Service rollback: stop keepalived only after confirming peer holds VIP. Emergency stop: if Keepalived conflicts with network gear, stop it, but ensure VIP not needed elsewhere.

7.3 Emergency Mitigation

VIP moved but traffic fails: verify VIP on new node, Nginx listening 0.0.0.0, local curl works, upstream reachable, switch MAC table correct. If local works but external fails → ARP convergence; run arping -c 5 -A -I eth0 VIP.

Dual-master: immediately stop keepalived on lower-priority node (confirm priority first). After restore, fix heartbeat, restart stopped node, verify single MASTER.

Config drift after failover: scp configs from old master, nginx -t && nginx -s reload. Root fix: config management via template+CI; add config diff check in notify script.

Daily Maintenance

8.1 Routine Checks

Daily: VIP on exactly one node (scripted check), keepalived/nginx active, health-check log FAIL count. Weekly: tcpdump VRRP ads, config diff, version match, cert expiry. Monthly: full failover drill (low traffic, announced), verify preempt_delay / nopreempt still appropriate, check GARP behavior after network changes.

8.2 Drill Methodology

Drill must measure business-side impact, not just VIP movement. Run continuous requests during drill, record success rate and max outage. Expected failures ~10-20 (5-10s outage at 0.5s interval). Log three numbers: first failure timestamp, recovery timestamp, total failures — these are your measured SLA.

8.3 Upstream Routing Coordination

If cloud LB or DNS sits above VIP, their health-check intervals (5-30s) may be slower than VRRP failover, causing additional 10-30s outage. Include this in postmortem. DNS TTL should be low (60s) but cannot eliminate cache window.

Troubleshooting Cheatsheet

Both MASTER: heartbeat down, VRID mismatch, unicast_peer wrong → tcpdump -i eth0 'ip proto 112' → fix heartbeat, verify IDs/peers

VIP moved but ping fails: GARP dropped, switch ARP suppression → check switch MAC table → arping -A, increase GARP repeats

Stop nginx, no failover: weight insufficient (priority+weight ≥ peer) → calc priority+weight vs peer → weight ≤ -(diff+10)

Config drift after failover: manual edit on one node → diff -r configs → config management

Flapping: NIC flap, health-check too sensitive → journald failover frequency → preempt_delay, increase fall Master reclaims VIP: missing nopreempt or ignored (state MASTER) → check for "nopreempt is useless" warning → state BACKUP + nopreempt or preempt_delay Keepalived won't start: vrrp_strict + authentication conflict, script perms → keepalived -t → fix per error

Script not running: enable_script_security permission denial → journalctl | grep SECURITY → chown root:root && chmod 755 VIP exists but port unreachable: Nginx bound to VIP not 0.0.0.0 → ss -lntp | grep :80 → change to listen 80 Verification methods matter more than fixes — confirm root cause before changing config.

Summary

Three pillars for deterministic HA: (1) Correct failover condition — health-check validates serviceability not just process; priority+weight arithmetic must actually trigger failover. (2) Verified failover process — test four scenarios (process crash, node crash, recovery behavior, heartbeat loss), record measured outage. (3) Post-failover convergence — split-brain unavoidable but must auto-converge; preempt_delay balances stability and self-healing. Documentation must capture current topology, measured failover impact, manual takeover steps, and change log with rationale.

Extended Scenarios

12.1 Multiple VIPs in One Instance vs Multiple Instances

Multiple vrrp_instance with separate VIPs elect independently — can cause VIPs split across nodes. If sites should fail together, use single instance with multiple virtual_ipaddress entries. If independent failover needed, use distinct VRIDs, priorities, and document VIP ownership.

12.2 vrrp_sync_group

Binds multiple instances to fail together: one FAULT → whole group FAULT. Use for internal+external VIP pairs that must stay on same node. Group notify only, not per-instance; priorities should match; one instance per group.

12.3 Layer 4 + Layer 7 Coexistence

Separate vrrp_script for each ( chk_nginx for HTTP, chk_stream for TCP port). Independent instances with different VRIDs and VIPs so L7 failure doesn't affect L4. Stream check uses ss -lnt and bash /dev/tcp for handshake verification.

12.4 VIP Failover & conntrack

Stale conntrack entries on old node or intermediate firewalls can cause connections to hang. conntrack -D -d VIP deletes entries (breaks those connections) — use only to accelerate failure detection when connections already dead. Better: application-level timeouts/retries.

Cloud Environment Differences

13.1 No Multicast

Cloud VPCs block multicast (224.0.0.18). Must use unicast mode: unicast_src_ip and unicast_peer both required.

13.2 Source/Dest Check

Cloud NICs enforce source/dest check; VIP traffic dropped unless disabled via console/API (AWS: modify-network-interface-attribute --no-source-dest-check).

13.3 Cloud-Native Alternatives

Cloud LB + health checks simpler for basic failover. Keepalived needed only for source IP preservation or vendor independence. Cross-AZ: use per-AZ VIPs with DNS/LB distribution; VIP cannot float across subnets.

Final Word

HA value lies in deterministic failure behavior, not config complexity. Achieve via correct conditions, verified drills, and guaranteed convergence. Invest in checklists, regression scripts, and measured outage data — they outweigh fancy configs. HA isn't about zero failures; it's about predictable behavior when failures happen.

Appendix: Deployment Checklist & Command Reference

12-item pre-flight checklist (VIP free, VRRP allowed, heartbeat reachable, configs identical, keys match, syntax ok, script perms, health endpoint 200, Nginx listens 0.0.0.0, time sync, single VIP holder, measured failover <10s). Common commands for status, manual intervention, forced failover, troubleshooting. Regression script automates post-change validation (services up, health 200, VIP on exactly one, VRRP peer role). Documentation essentials: current topology, measured failover impact, manual takeover steps, change log with reasons.