Skip to content
Keepalived VRRP HA: Brain Split, Priority Tuning, and Business-Aware Health Checks
Operations 52 min read

Keepalived VRRP HA: Brain Split, Priority Tuning, and Business-Aware Health Checks

This article analyzes Keepalived VRRP high availability, detailing brain split causes, priority/preempt trade-offs, business-aware health checks, switch time breakdown, cloud multicast limitations, production configurations with unicast, source code insights, and operational practices for reliable failover.

Architecture Pony
Architecture Pony
Architecture Pony

1. Technical Analysis: Four Key Aspects of VRRP

1.1 VRRP Protocol: Master-Backup Maintained by Advertisements

VRRP's core mechanism is straightforward: a group of routers share a virtual IP (VIP) and a virtual MAC (00:00:5E:00:01:VRID). Only the MASTER answers ARP and forwards traffic. The MASTER sends an advertisement every advert_int (default 1 second) to multicast address 224.0.0.18, protocol 112, announcing "I'm alive." BACKUP nodes reset their timers upon receiving advertisements; after missing three consecutive advertisement intervals (3 × advert_int + skew), they assume the MASTER is dead and take over. Thus VRRP fault detection is essentially a heartbeat timeout, with precision controlled by advert_int — smaller values detect faster but increase false positives from network jitter.

Priority (0–255) determines the master: higher wins. Value 255 is reserved for the "IP owner" (the VIP is natively configured on that interface), which becomes MASTER unconditionally. When priorities are equal, the higher real IP wins — a fallback rule many overlook. The priority gap must be large enough (at least 20) because health-check scripts typically deduct weight; a small gap can be flipped by a single deduction (as in Incident 2).

1.2 State Machine: Three States and Two Transitions

Keepalived's VRRP instance has three states: INIT (starting, interface not ready), BACKUP (waiting for advertisements), MASTER (holds VIP, sends advertisements, answers ARP). Two transition rules: BACKUP becomes MASTER if no advertisement is received within

master_down_interval</sub>; MASTER falls back to BACKUP if it receives an advertisement with higher priority <em>and</em> preemption is enabled (<code>preempt

). preempt is enabled by default — the direct cause of Incident 2: a brief misjudgment promotes BACKUP to MASTER; when the original MASTER resumes advertisements, it preempts the VIP back, causing two business disruptions per flap. Production practice: configure both nodes as state BACKUP with explicit nopreempt, letting priority decide mastership only at startup.

1.3 Brain Split: Three Causes and One Detection Command

Brain split (dual MASTER) is Keepalived's most severe failure, with only three root causes:

Multicast blocked — switch ACL/VLAN isolates 224.0.0.18, or firewall blocks protocol 112 (VRRP is a separate IP protocol; many only allow TCP/UDP).

Cloud environments lack multicast — most public cloud VPCs disable multicast; must switch to unicast ( unicast_peer) or abandon VRRP.

Configuration mismatch — differing virtual_router_id, auth_pass, or advert_int creates two distinct VRRP groups, inherently dual MASTER.

Detection is a single command on both nodes: ip addr | grep VIP — simultaneous presence equals dual MASTER, no exceptions. Add this to pre-deployment inspection scripts; cost is zero.

Once detected, mitigation order: protect business consistency first, then investigate . For stateless services (e.g., plain Nginx), dual MASTER merely splits traffic; tolerate briefly while fixing the network. For stateful or write-heavy services, immediately stop keepalived on the lower-priority side ( systemctl stop keepalived) — better a single point than dual writes. This is why a "quorum" check belongs in the health script: ping the gateway or a third-party address; if unreachable (local network partition), voluntarily relinquish the VIP — better to go down yourself than create a split-brain .

1.4 Health Checks: Criteria, Weight, and Timing Parameters

Health-check script criteria priority (highest to lowest):

Business availability — curl a real endpoint and verify HTTP status code.

Port reachable — nc or curl root path.

Process exists — ps.

Lower criteria are easier to fake: a process may exist while the service is broken (Incident 3). The script's exit code is the sole contract: 0 = healthy, non-zero = failure. The script must enforce its own timeout (e.g., curl -m 2), otherwise a hung check blocks Keepalived's scheduling thread. Three timing parameters govern detection sensitivity: interval — execution period. fall — consecutive failures before declaring fault. rise — consecutive successes before declaring recovery.

Common combo: interval 2 / fall 3 / rise 2 → fastest fault detection 6 seconds, slowest recovery confirmation 4 seconds. This is the lower bound of failover latency; never promise "sub-second failover."

On script failure, two handling modes:

Weight deduction (e.g., weight -30) lowers effective priority; if it drops below the peer, failover occurs naturally. Suits "degraded but still serving" intermediate states.

Direct FAULT state — script returns non-zero without weight configured; Keepalived immediately places the instance in FAULT and surrenders VIP. Faster, more decisive, but demands extremely reliable criteria.

Trade-off: weight is gentler, recoverable, avoids flapping on a single glitch; FAULT is decisive but requires bulletproof checks. Production recommendation: weight + sufficiently large priority gap (e.g., 150 vs 100) balances stability and sensitivity.

1.5 Failover Time Accounting: Five Segments

A complete failover comprises five additive segments:

Detection latency = interval × fall (default 2×3 = 6 s).

Script timeout — if failure manifests as timeout, add the -m value.

VRRP convergence = 3 × advert_int (~3 s).

Gratuitous ARP broadcast & switch MAC table update (ms to s, device-dependent).

Client reconnect & DNS/pool refresh (seconds to minutes, depends on client keepalive/retry).

Total 8–15 seconds is typical. Pushing below 5 seconds requires shrinking interval / fall / advert_int, which raises false-positive rates. Hence HA designs must include two safety nets: client-side retries and upstream eviction — VIP failover always has a window; the application layer cannot assume it's instantaneous.

1.6 Cloud Environments: Ask If Multicast Works First

This is the first architectural fork: public cloud VPCs generally lack multicast support , so default VRRP fails (symptom: dual MASTER). Three paths:

Switch to unicast — unicast_src_ip + unicast_peer with peer's real IP; UDP unicast replaces multicast, works on cloud and is simplest.

Use cloud load balancer — SLB/CLB multi-AZ deployment; offload HA to the provider; Nginx becomes stateless nodes.

DNS primary/backup or multi-value — short TTL, health-check integration; failover slower (TTL-bound) but network-layer independent.

Decision order: if SLB works, don't self-host VIP ; if self-host is mandatory (cost, compliance, internal network), prefer unicast VRRP; worst choice is lifting the data-center multicast design to the cloud unchanged and spending a week debugging "why always dual MASTER."

2. Core Workflows: Configuration, Failover, and Detection

2.1 Production Config 1: Keepalived Primary/Backup (with Unicast)

# /etc/keepalived/keepalived.conf
global_defs {
  router_id nginx_a  // backup changes to nginx_b
  script_user root
  enable_script_security
}

vrrp_script chk_nginx {
  script "/etc/keepalived/chk.sh"
  interval 2  // probe every 2s
  fall 3      // 3 consecutive failures = fault
  rise 2      // 2 consecutive successes = recovery
  weight -30  // deduct 30 on failure
}

vrrp_instance VI_1 {
  state BACKUP  // both nodes BACKUP
  interface eth0
  virtual_router_id 51  // must match
  priority 150  // backup 100, gap must be large
  advert_int 1
  nopreempt  // disable preemption, prevent flap

  unicast_src_ip 10.0.0.11  // local
  unicast_peer {
    10.0.0.12  // peer
  }

  authentication {
    auth_type PASS
    auth_pass NginxHa2026
  }

  virtual_ipaddress {
    10.0.0.100/24 dev eth0
  }

  track_script {
    chk_nginx
  }

  notify_master "/etc/keepalived/nt.sh master"
  notify_backup "/etc/keepalived/nt.sh backup"
  notify_fault  "/etc/keepalived/nt.sh fault"
}

Every non-default value prevents a specific pitfall:

Both state BACKUP — avoids "role hardcoded" causing instant dual MASTER at boot; priority alone decides.

nopreempt — prevents Incident 2's VIP ping-pong.

priority 150 vs 100 — leaves 30 points headroom (after weight -30 effective priority 120 still > 100), so a single script failure won't flip mastership.

unicast — mandatory for cloud or cross-subnet; also easier to pass firewall audits.

Three notify hooks — emit alerts, update upstream registration, clear connection pools; entry point for "observable failover." Note: auth_pass only uses first 8 characters; don't obsess over length.

2.2 Production Config 2: Health Check & Brain-Split Arbitration Script

#!/bin/bash
# /etc/keepalived/chk.sh
# 1) Business availability: real endpoint + status code
CODE=$(curl -s -o /dev/null -m 2 \
  -w "%{http_code}" \
  http://127.0.0.1/health)
[ "$CODE" = "200" ] || exit 1

# 2) Brain-split arbitration: can I reach the network?
ping -c 1 -W 1 10.0.0.1 >/dev/null 2>&1 \
  || exit 1

# 3) Optional: ensure not dual MASTER
CNT=$(ip addr | grep -c 10.0.0.100 2>/dev/null || true)
[ "$CNT" -le 1 ] || exit 1

exit 0

The script's three sections map to three criteria:

Business availability — curl /health must return 200; prevents "process up but service dead."

Network reachability — ping gateway failure means local network isolation; must not hold VIP (would become "island master").

Non-dual-MASTER — local VIP count > 1 indicates another holder; voluntarily exit.

Total runtime hard-capped at ~3 seconds by curl -m 2 and ping -W 1, preventing Keepalived scheduler starvation. The script itself does not manage state or restart services — restarting Nginx is a separate concern; mixing it into the check creates a "check fails → restart → check fails" oscillation loop. Correct approach: delegate self-healing to systemd's Restart policy.

2.3 Failover Timeline: From Fault to Traffic Recovery

Walk through chronologically (using interval 2 / fall 3 / advert_int 1):

T1: Primary Nginx workers crash; /health starts returning 502.

T2: Next check fails; after three consecutive failures (~6 s) script failure count reaches threshold.

T3: track_script takes effect; primary priority 150 minus 30 = 120. Is 120 lower than backup's 100? No — so no failover occurs here. This arithmetic is where many configs go wrong: weight deduction must actually drop below the peer to trigger failover; either narrow the priority gap, increase deduction, or switch to FAULT mode.

T4: Primary stops answering ARP for VIP (enters FAULT, ceases advertisements).

T5: Backup misses advertisements for 3 seconds, transitions to MASTER.

T6: Backup sends gratuitous ARP; switch MAC table updates; VIP traffic redirects.

T7: Clients reconnect successfully.

Key lesson: the T3 arithmetic must be calculated before deployment — a 50-point gap with -30 weight never fails over; such a config "looks like it has health checks" but is actually a placebo.

2.4 Five Pre-Launch Validations

Configuration changes must pass five gates before going live:

Syntax — keepalived -t -f validates config; nginx -t likewise.

Single MASTER confirmation — both nodes run ip addr | grep VIP; only one may have it.

Planned failover — systemctl stop keepalived on primary; observe backup takeover time, business interruption, and whether failback occurs (with nopreempt it should not).

Health-check stress test — make /health return 500 or stop Nginx; verify VIP actually moves. Catches "weight math wrong so no failover."

Brain-split drill — iptables drop VRRP traffic to simulate multicast loss; verify arbitration script forces one side to yield.

All five must pass to consider launch complete ; skipping any guarantees the flaw will surface spectacularly during a real incident.

2.5 Notify Hooks: Turning Failover into Observable Events

The notify_master/backup/fault hooks are the only programmable entry points during state transitions. They perform four actions:

Alert (DingTalk/WeCom, include hostname and current VIP).

Timestamp — write to local file for post-mortem failover window measurement.

Notify upstream — add/remove self from service registry/gateway backend list; closes the "VIP failover + application awareness" loop.

Clear connection pools — reload/restart Nginx to avoid stale connections pointing to dead upstreams.

Hook scripts must be idempotent and fast — they run on Keepalived's state-transition path; a blocked HTTP callback directly slows failover. Common production pattern: hooks only write a local state file and publish a message; heavy lifting is done asynchronously by an external agent.

2.6 Node Consistency: The Most Overlooked HA Prerequisite

The easiest HA oversight isn't the protocol — it's whether the two nodes' "content" is equivalent . Failover moves only the VIP, not nginx.conf, certificates, static assets, or upstream lists. Failing over to a node with missing config upgrades the outage from "service unavailable" to "service returns wrong results," which is harder to debug. Four content categories must be managed:

Configuration ( nginx.conf + conf.d) — in Git, rendered from unified templates, deployed via single pipeline.

Certificates (expiry, chain, private key permissions) — both nodes use the same signed artifact.

Static assets & frontend builds — must live on object storage or shared NFS; never rely on per-node rsync that drifts.

Upstream & backend lists — if dynamic, both connect to same registry; if static, generated from same source file.

Validation is low-tech but effective: randomly reboot one node; business must remain fully functional. This exposes 90% of consistency defects and should be a monthly routine.

3. Core Source Code: Four Critical Logic Sections

3.1 State Machine Decision: One Priority Comparison Decides Fate

// vrrp_state_backup(), upon receiving advertisement
if (hd->priority == 0 ||
    hd->priority < vrrp->effective_priority ||
    (hd->priority == vrrp->effective_priority
     && inet_cmp(hd->ip, vrrp->ip))) {
  // Peer priority lower, or equal but IP smaller
  // and local wants MASTER -> preempt
  if (vrrp->wantstate == WANT_MASTER) {
    vrrp_state_master_tx(vrrp);
  }
}

This snippet explains three behaviors:

Priority 0 is special — advertisement carrying 0 means "voluntarily relinquish MASTER"; peer should take over immediately. This is why systemctl stop keepalived fails over quickly.

effective_priority is the real-time value after deductions , not the static config value; thus weight deductions genuinely participate in comparison.

Equal priority compares IP — larger IP wins, guaranteeing a deterministic master even when configs are identical, avoiding permanent deadlock.

wantstate is controlled by preempt — with nopreempt, wantstate stays BACKUP; even if local priority is higher, it won't preempt. This is the anti-flap mechanism.

3.2 Script Execution & Weight Update

Scripts are forked periodically by Keepalived's dedicated scheduler thread. Upon return code, two actions occur: update failure/success counters, decide whether to adjust weight or state. Three design points:

Script must self-timeout — if the forked child hangs, Keepalived can only wait; the entire check chain stalls. Hence curl -m / ping -W are mandatory.

enable_script_security + script_user — since v1.3, default denies root execution; missing these is the #1 reason "configured script doesn't run" (logs show only a warning).

Counters are consecutive — fall 3 means three consecutive failures; a single success resets the counter. Thus occasional jitter won't trigger failover, which is also why interval cannot be too small (too small makes it easier to accumulate consecutive failures).

3.3 Gratuitous ARP: The Step That Actually Redirects Traffic

After binding the VIP, the new MASTER must do one thing: send gratuitous ARP, proactively telling the network "this IP is now at this MAC." Without it, switch MAC tables and upstream ARP caches still point to the old node; traffic continues to the failed node — the classic "VIP moved but business didn't recover." Keepalived sends it by default (via gratuitous_arp params), but some cloud networks or ARP-protection environments drop it, requiring explicit arping supplementation. Diagnostic commands: arping -I eth0 -c 3 VIP and upstream arp -n — faster than log diving.

3.4 Notify Hook Execution Order

Hooks run after the state machine completes transition: state changed → VIP bound/released → execute notify . Therefore notify_master can safely assume VIP is local and perform "register with upstream" actions. Conversely, slow hooks don't block the failover itself (transition already done), only delay subsequent actions. This ordering dictates practice: alerting belongs in hooks (accurate), health checks must not rely on hooks (latency) . Another detail: hooks run as root with minimal environment (PATH may lack /usr/sbin); absolute paths for all commands are mandatory.

4. Advanced Topics: Seven Accounts You Must Calculate Upfront

4.1 Primary/Backup vs. Active/Active: Calculate Resource Utilization First

Active/standby costs half the resources idle year-round ; benefit: simple failover logic, clear state, isolated fault domain. Active/active doubles utilization but introduces three new problems: session stickiness, cache inconsistency, and "when one fails, the other must instantly absorb double traffic." Decision criterion: can a single node handle peak×2? If a node already runs >60% at peak, active/active guarantees avalanche on failure; primary/backup is safer. Small/medium scale (ample single-node headroom) → primary/backup; large scale → active/active + elastic scaling, with mandatory load test of "single node handling double."

4.2 Multi-Level HA: LVS + Keepalived + Nginx

Classic four-layer stack stacks two HA layers: LVS (DR mode) does L4 load balancing; Keepalived provides VIP failover for LVS; behind them, a pool of Nginx handles L7. Value: LVS forwards in kernel space, throughput far exceeds Nginx; Nginx pool scales horizontally to dozens of nodes, single-node failure is invisible. Cost: operational complexity rises a notch (LVS DR mode requires backends in same subnet and ARP suppression — the #1 rookie trap). Adoption threshold: only when QPS exceeds a single Nginx's capacity ; otherwise the extra layer is self-inflicted pain.

4.3 Cloud Alternatives: SLB, DNS, Anycast

Three cloud-native approaches compared:

SLB/CLB — multi-AZ, health checks, elastic bandwidth fully managed; failover guaranteed by provider. Downside: cost, L4/L7 capabilities limited by product.

DNS primary/backup — TTL 30–60 s; failover slow but works across clouds/regions; suits disaster recovery, not high availability.

Anycast — same IP announced in multiple locations; routing layer auto-converges on nearest/healthy. Strongest but highest barrier (requires BGP + own IP block).

Pragmatic choice: same cloud → SLB; cross-cloud DR → DNS; own BGP + IP block → consider Anycast. Self-hosted Keepalived only wins for "internal network, cost-sensitive, or special network policy" scenarios.

4.4 Failover Drills: No Drill = No HA

Quarterly drills, four steps:

Planned failover — stop keepalived; record failover duration, error rate, upstream awareness.

Fault injection — kill -9 Nginx workers or iptables DROP port; verify script criteria actually trigger.

Brain-split simulation — block VRRP traffic; verify arbitration logic forces one side to yield.

Failback verification — restore primary; confirm nopreempt holds, no ping-pong.

Drills must log "actual failover seconds" and compare against SLA — config tweaks (e.g., changing interval from 2 to 5) silently widen the window, and nobody recalculates. Drill's greatest value isn't proving success; it's discovering "thought it was configured but it wasn't effective" — the kind that kills you during real incidents.

4.5 Monitoring: Three Must-Alert Metrics

VRRP state changes — any master/backup transition alerts (even successful ones); frequent flapping signals network instability.

Dual MASTER detection — poll both nodes' VIP ownership every minute; simultaneous true = P0 alert. This is Incident 1's only safety net.

Health-check script duration — script approaching timeout indicates local load anomaly; precursor to failure.

All three are "zero cost, high yield"

: one curl + one ip addr collects them, yet they save hours of debugging in incidents like three-hour undetected dual MASTER. Monitoring's value here is stark: not "is business dead?" but "is the HA you think you have actually there?"

4.6 Six High-Frequency Pitfalls

Both nodes state MASTER — instant dual MASTER at boot.

Priority gap too small or weight deduction insufficient — health check becomes decorative (see §2.3 arithmetic).

Firewall blocks protocol 112 — only TCP/UDP allowed; VRRP is a separate IP protocol.

enable_script_security not set — script never runs; log shows only a warning.

VIP failover without upstream notification — gateway keeps sending traffic to old node (§2.5 notify loop).

Backup missing identical Nginx config — failover lands on broken node, immediate 404. Most ironic: HA fails over to a broken machine.

4.7 Who Covers the 8-Second Window: Application-Side Four-Piece Set

VIP failover's 8–15 second window cannot be eliminated; the application layer must absorb it. Four essentials, none optional:

Connection retry — clients retry connection-class errors (refused/reset) with bounded attempts covering the window (e.g., 3 retries, 1 s interval, 5 s total budget). Only idempotent endpoints may retry writes.

Short connect timeout — set 1–2 s, not default 30 s; otherwise a single failover stalls the entire thread pool.

Connection pool health checks — post-failover, all pooled connections are dead; must enable validate-on-borrow or test-on-idle to avoid continuously handing out dead connections.

Fast fail + circuit breaker — error spike during failover must trigger rapid circuit breaking and fallback page, not tie up all threads waiting.

These four turn "HA failover" from a user-visible outage into a monitoring blip — that's the true completion of HA. Only configuring Keepalived without application-side safety nets merely "shortens outage duration"; it doesn't achieve "high availability."

5. Learning Path: From Configuring to Daring to Failover

Step 1: Build a two-VM lab — apply §2.1 config, verify single MASTER with ip addr, manually stop keepalived once, feel the 8–15 second window firsthand. Step 2: Write the check script "right" — first use ps version to create "process up but service dead"; watch it not failover. Then switch to curl /health version; watch it failover. This A/B comparison beats reading ten articles. Step 3: Read VRRP state machine & advertisement handling source — understand priority 0, equal-priority IP comparison, wantstate; then read keepalived.conf man page for precise semantics of weight, fall, rise, nopreempt. Step 4: Run a full brain-split drill and cloud alternative comparison (SLB vs. self-hosted VIP cost/capability matrix) to build judgment on "should I self-host?"

6. High-Frequency Interview Questions (with Answers)

Q1: How does Keepalived work? Based on VRRP: nodes share VIP + virtual MAC; MASTER sends advertisements to 224.0.0.18 every second; BACKUP takes over after missing 3 consecutive intervals and broadcasts gratuitous ARP. Master election by priority (higher wins, 255 = IP owner), tie-break by IP. Essence: heartbeat timeout + priority comparison.

Q2: What is brain split? Causes, detection, prevention? Both nodes think they're MASTER, both bind VIP. Causes: multicast blocked by ACL/VLAN, firewall blocks protocol 112, mismatched virtual_router_id / auth_pass, cloud lacks multicast. Detection: both run ip addr | grep VIP; simultaneous presence = dual MASTER. Prevention: unicast unicast_peer, arbitration script (ping gateway failure → relinquish VIP), dual MASTER monitoring alert.

Q3: Why recommend both nodes state BACKUP + nopreempt ? Avoids "hardcoded role" causing instant dual MASTER at boot; lets priority solely decide. nopreempt disables preemption: after backup takes over, original master's recovery won't snatch VIP back, preventing network-jitter-induced VIP ping-pong (each flap causes seconds of outage). Trade-off: no auto-failback after recovery; requires manual or planned operation.

Q4: What constitutes VRRP failover time? Detection latency ( interval × fall) + script timeout + VRRP convergence (3 × advert_int) + gratuitous ARP + MAC update + client reconnect. Default totals 8–15 s. Faster requires shrinking those three parameters, but false positives rise. Hence application layer must provide retry and upstream eviction safety nets.

Q5: What should health check script examine? Business availability: curl local /health for HTTP 200, with -m 2 timeout. Not just process (master up but workers dead), not just port (port open but returns 502). Script must self-timeout or it blocks Keepalived scheduler. Self-healing belongs to systemd, not the check script.

Q6: Meanings of weight , fall , rise ? weight = priority deduction on script failure (e.g., -30); failover only if deducted priority drops below peer. Without weight, script failure directly sets instance to FAULT, surrendering VIP. fall = consecutive failures to declare fault; rise = consecutive successes to declare recovery. Together they tune sensitivity vs. anti-flap.

Q7: Why does weight -30 never trigger failover? Arithmetic: priority 150 − 30 = 120, still > peer's 100 → no failover. Health check becomes a prop. Either shrink priority gap to ≤30, increase deduction, or switch to FAULT mode. Must manually verify this inequality before launch.

Q8: What is gratuitous ARP? Consequence of skipping? New MASTER broadcasts "VIP now at this MAC" to instantly update switch MAC tables and upstream ARP caches. Without it, traffic continues to old node → "VIP moved but business didn't recover." Cloud networks or ARP protection may drop it; supplement with arping and verify.

Q9: Why frequent dual MASTER on cloud? Solution? Most VPCs lack multicast; VRRP advertisements never arrive; both sides miss heartbeats. Fix: switch to unicast ( unicast_src_ip + unicast_peer), use cloud SLB, or DNS primary/backup. Worst: lift data-center multicast design to cloud unchanged.

Q10: Notify hooks purpose and limits? Run after state transition; used for alerting, timestamping, upstream registration/deregistration, connection pool clearing. Limits: run as root, minimal env (absolute paths required), on transition path so must return fast and be idempotent. Best practice: hooks only write local state file + emit message; heavy actions async via external agent.

Q11: Special meaning of priority 255? 255 = IP owner: VIP natively configured on this interface; unconditionally becomes MASTER and always sends advertisements. Used for "primary preferred" scenarios. Immune to weight deductions, so health checks ineffective against it — a design trade-off.

Q12: Advertisement priority 0 meaning? Voluntary MASTER relinquishment. Receiving BACKUP takes over immediately, no heartbeat wait. This is why systemctl stop keepalived fails over fast; standard for planned maintenance (more graceful than kill -9).

Q13: Primary/backup vs. active/active selection? Check if single node has headroom for double traffic. Peak >60% → must use primary/backup (active/active would avalanche on failure); ample headroom → active/active for utilization, but must solve stickiness, cache consistency, and must load-test "single node handling double." Utilization isn't the only axis; fault domain and complexity count too.

Q14: Value of LVS+Keepalived+Nginx four-layer stack? LVS forwards in kernel, throughput far above Nginx; Keepalived gives LVS VIP failover; Nginx pool behind scales horizontally, single-node failure invisible. Cost: complexity (LVS DR mode requires same subnet + ARP suppression — top rookie trap). Only justified when single Nginx capacity is exceeded.

Q15: How to verify HA config actually works? Five gates: keepalived -t + nginx -t syntax; both nodes ip addr confirm single MASTER; planned stop measures failover time & business impact; corrupt /health to 500 verifies script triggers failover (catches weight math bugs); iptables block VRRP for brain-split drill. All five must pass; quarterly re-drill.

Q16: Failover done but traffic not recovered — where to look? Sequence: was gratuitous ARP sent? (verify with arping); upstream ARP/MAC caches updated?; backup's Nginx config & static assets identical to primary? (often overlooked, failover → 404); upstream connection pools pointing to dead nodes?; DNS or client pools still referencing old IP?

Q17: What does enable_script_security do? Keepalived ≥1.3 security: requires scripts run as non-root with file permissions preventing writes by others. Must set script_user; otherwise script execution is refused, log shows only a warning — #1 cause of "configured check but not effective."

Q18: First action when dual MASTER occurs? Stop bleeding, then investigate: stateless (pure Nginx proxy) can tolerate briefly while fixing network; stateful/write-heavy must immediately stop keepalived on lower-priority side — better single point than dual write. Then check multicast reachability ( tcpdump protocol 112), firewall, virtual_router_id / auth_pass consistency.

Q19: Why ping gateway in check script? Brain-split arbitration: ping failure means local node isolated from outside; even if local Nginx healthy, must not hold VIP — otherwise becomes "island master" (traffic in, no traffic out) and may form dual MASTER with peer. Better to voluntarily surrender VIP, let healthy side carry load.

Q20: Cloud self-hosted VIP vs. SLB — how to choose? Same cloud → prefer SLB: multi-AZ, health checks, elastic bandwidth fully managed, failover guaranteed by provider, avoids multicast/brain-split pitfalls. Self-hosted VIP fits internal network, cost-sensitive, or special L7 needs; must use unicast VRRP. Cross-cloud DR → DNS primary/backup. Own BGP + IP block → consider Anycast.

7. One-Page Cheat Sheet

Keepalived Cheat Sheet: Protocol : VRRP, multicast 224.0.0.18 protocol 112, advert 1s, miss 3 → takeover. Election : priority higher wins, 255 = IP owner, tie → higher IP. Anti-flap : both state BACKUP + nopreempt + priority gap ≥ 30. Immobile arithmetic : priority gap must be < weight deduction, else check is decorative. Script criteria : curl /health 200 > port > process, must have -m timeout. Brain-split detection : both nodes ip addr | grep VIP, simultaneous = dual MASTER. Failover window : detect 6s + converge 3s + ARP + reconnect ≈ 8–15s. Cloud : multicast often unavailable → unicast or SLB. Three alerts : state change, dual MASTER, script latency.

8. Summary & Next Episode Preview

HA's value isn't "configured Keepalived" but that the failover event has been validated, monitored, and calculated . Recap:

Multicast is VRRP's lifeline — when it breaks, dual MASTER appears; hence arbitration and dual MASTER alerts are mandatory.

Health check criteria must be business-level — ps fools humans, curl doesn't.

Failover time is calculable — 8 to 15 seconds window must be covered by application-layer retries, not hope.

Next episode enters programmable gateway: OpenResty + Lua — why Nginx config language falls short, how cosocket achieves non-blocking, and writing dynamic logic like rate limiting, canary, auth in dozens of lines of Lua.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

OperationsHigh AvailabilitynginxFailoverVRRPHealth CheckKeepalivedCloud HABrain Split
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.