Master LVS Load Balancing: Three Modes and Production‑Grade HA Deployment Guide

What is Load Balancing?

Load balancing distributes massive requests across multiple servers to increase overall system performance, similar to adding more checkout counters in a fast‑food restaurant.

Hardware Load Balancers

Typical hardware solutions include F5 BIG‑IP and A10 Thunder. F5 offers extensive features (SSL offload, WAF, health checks) but costs hundreds of thousands of dollars, targeting financial‑grade customers. A10 provides similar performance at 30‑50% lower price, making it a cost‑effective alternative.

Linux Virtual Server (LVS)

LVS, an open‑source project started by Dr. Zhang Wensong in 1998, is integrated into the Linux kernel and provides kernel‑level load balancing. Clients access a single virtual IP (VIP) without knowing the number of real servers behind it.

Core Components

Director : receives client requests and selects a Real Server based on a scheduling algorithm.

VIP : the public virtual IP address.

Real Server (RS) : the backend server that actually processes the request.

Three LVS Working Modes

1. NAT Mode

Client → Director (VIP) → RS (packet IP rewritten). The response returns to the Director, which rewrites the source IP back to the VIP before sending it to the client. Both request and response pass through the Director, making it a bottleneck for large responses. Suitable for small‑scale deployments due to its simplicity.

2. DR Mode (Direct Routing)

The Director only rewrites the destination MAC address; the IP address remains unchanged. After the Director forwards the packet, the RS processes it and sends the response directly to the client, bypassing the Director. This reduces load on the Director and can achieve 80‑90% of hardware‑level performance. DR requires the Director and all RSs to reside in the same LAN (or be L2‑connected).

3. TUN Mode (IP Tunneling)

To overcome DR’s LAN limitation, TUN encapsulates the original IP packet inside a new IP packet, allowing distribution across WAN links and multiple data centers. The encapsulated packet is sent to a remote RS, which decapsulates it, processes the request, and replies directly to the client. TUN supports multi‑region active‑active architectures but is more complex to configure.

LVS Management Commands (ipvsadm)

ipvsadm -L -n                     # List current LVS rules
ipvsadm -L -n --stats           # Show detailed statistics
ipvsadm -A -t 192.168.1.100:80 -s rr   # Add virtual service (TCP, port 80, round‑robin)
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.10:80 -g -w 1   # Add RS in DR mode
ipvsadm -a -t 192.168.1.100:80 -r 192.168.1.10:80 -m -w 1   # Add RS in NAT mode

Common parameters include -A (add virtual service), -a (add real server), -t / -u (TCP/UDP), -s (scheduling algorithm), -m (NAT), -g (DR, default), -i (TUN), -w (weight), and -p (persistence).

High‑Availability with Keepalived

Because a single LVS Director is a single point of failure, Keepalived (based on VRRP) provides HA by creating a virtual IP shared by a master and a backup Director. When the master fails, the backup instantly takes over the VIP.

Sample keepalived.conf (Master)

! Configuration File for keepalived

global_defs {
    router_id LVS_MASTER
    script_user root
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 123456
    }
    virtual_ipaddress {
        192.168.100.10
    }
}

virtual_server 192.168.100.10 80 {
    delay_loop 6
    lb_algo wrr
    lb_kind DR
    persistence_timeout 60
    protocol TCP

    real_server 192.168.100.21 80 {
        weight 3
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
            connect_port 80
        }
    }

    real_server 192.168.100.22 80 {
        weight 2
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

The backup node only changes state BACKUP, router_id LVS_BACKUP, and lowers priority to 90.

Real Server Setup for DR Mode

#!/bin/bash
VIP=192.168.100.10
GATEWAY=192.168.100.1

# Bind VIP to lo:0
ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP up
route add -host $VIP dev lo:0

# ARP suppression (critical)
echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce

# Enable forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

Without ARP suppression, IP conflicts cause RSs to answer directly, bypassing LVS and breaking load balancing.

Kernel Parameter Tuning for High Concurrency

# Network connection tracking
net.ipv4.ip_conntrack_max = 2000000
net.netfilter.nf_conntrack_max = 2000000
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

# Port range
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_tw_reuse = 1

# Memory and buffers
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Connection queues
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Apply changes with sysctl -p.

Starting Services

# Run ARP suppression script on all Real Servers
bash /root/rs_config.sh

# Enable and start Keepalived on Master and Backup
systemctl enable keepalived
systemctl start keepalived

# Verify LVS rules
ipvsadm -L -n

Common Troubleshooting

VIP not binding : check Keepalived logs and firewall for VRRP (UDP 112).

All traffic goes to one RS : verify weight settings with ipvsadm -L -n --stats.

DR mode ineffective : 99% of cases are missing ARP suppression; confirm with sysctl -a | grep arp.

Connection count not increasing : ensure net.ipv4.ip_conntrack_max is large enough.

High CPU on Director : consider switching to DR mode to offload return traffic.

Conclusion

Load balancing is essential for backend architectures. Hardware appliances like F5 and A10 suit financially unconstrained scenarios, while LVS combined with Keepalived offers a cost‑effective, high‑performance solution for most internet services. Among LVS modes, NAT is simplest but limited, DR delivers near‑hardware performance and is the production default, and TUN enables cross‑region active‑active deployments. Always deploy Keepalived for HA, configure ARP suppression for DR, and tune kernel parameters according to actual traffic.