Mastering LVS Load Balancing: Deploy LVS‑NAT & Keepalived with Real‑World Examples

LVS Load Balancer

LVS is an open‑source L4 TCP/UDP load balancer implemented as a Linux kernel module, known as Linux Virtual Server. It aims to provide a highly scalable, reliable, and manageable virtual server using load‑balancing and clustering technologies.

Deployment Architecture

Three‑layer architecture:

Load Scheduler Layer : entry point for client requests; must ensure high availability.

Server Cluster Layer : stateless backend servers for scalability.

Shared Storage Pool Layer : centralized data pool supporting statelessness.

Basic Concepts

Server Types :

DS (Director Server) : load‑balancing entry point, also called front‑end server.

RS (Real Server) : actual server handling traffic, also called back‑end server.

VS (Virtual Server) : logical entity composed of DS and RS, exposed to clients via a VIP.

IP Types :

VIP (Virtual IP) : address presented to clients.

DIP (Director IP) : DS address used to communicate with RS.

RIP (Real IP) : address of a real server.

CIP (Client IP) : client’s source address.

Software Architecture

LVS consists of two modules:

ipvsadm : user‑space CLI tool for managing clusters and real servers.

ipvs : kernel‑space forwarding module that works on the INPUT chain of iptables to perform L4 forwarding and scheduling.

Users add rules with ipvsadm, which are then enforced by ipvs.

Operating Principles

LVS‑NAT Mode

Features :

Client request destination IP is VIP, source IP is CIP, destination port is VPort, source port is random CPort.

Director Server selects a real server according to the LB policy.

L3 NAT rewrites destination IP to RIP (ingress) and source IP to VIP (egress).

L4 PAT rewrites destination port to RPort (ingress) and source port to VPort (egress).

Real server’s default gateway must point to DIP.

Drawbacks : DS becomes a bandwidth and performance bottleneck; limited elasticity of RS.

LVS‑TUN Mode

To avoid frequent IP/Port editing, TUN mode introduces IP tunneling, creating a transparent path where client packets are encapsulated and not modified. All RSs must bind the VIP, allowing them to reply directly to the client, forming a triangular traffic flow and increasing throughput up to tenfold compared to NAT.

Features :

Client request is encapsulated in an outer IP tunnel; no packet editing.

Does not support PAT.

RSs bind the same VIP on a loopback interface.

RS default gateway is the front‑end router; responses bypass DS.

Drawbacks :

RS must support IP tunneling, which is intrusive.

Additional encapsulation overhead.

Higher design and deployment complexity.

LVS‑DR Mode

DR (Direct Routing) is similar to TUN but operates at L2, eliminating tunneling overhead and avoiding RS intrusion. DS and RS must reside on the same LAN, and DS rewrites the destination MAC to forward packets directly to RS.

Features :

DS and RS share the same L2 LAN.

All RSs configure a loopback VIP and suppress ARP replies.

DS rewrites destination MAC to RS‑MAC; RS replies directly to client.

DS default gateway is the front‑end router; no PAT support.

Drawbacks : Requires same LAN, limiting cluster size.

Load‑Balancing Algorithms

Static Algorithms

rr (Round Robin) : distributes requests evenly without tracking state.

wrr (Weighted Round Robin) : assigns weights based on server capacity.

sh (Source Hashing) : session affinity by hashing client IP.

dh (Destination Hashing) : similar to sh but hashes destination IP, often used for cache servers.

Dynamic Algorithms

lc (Least Connections) : sends new connections to the server with the fewest active connections.

wlc (Weighted Least Connections) : combines least‑connections with server weight.

sed (Shortest Expected Delay) : prefers servers with higher weight but may cause imbalance.

nq (Never Queue) : improvement over sed; assigns to idle servers directly.

LBLC (Locality‑Based Least Connections) : keeps same destination IP traffic on the same server to improve cache locality.

LBLCR (Locality‑Based Least Connections with Replication) : similar to LBLC with dynamic replication of server groups.

ipvsadm CLI

View Help

$ ipvsadm --help

List Rules

$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port   Forward Weight ActiveConn InActConn
UDP  10.0.0.9:5678 rr
  -> 192.168.1.8:5678   Masq    1       0          0

Add Cluster Service

$ ipvsadm -A -t 192.168.1.100:80 -s rr

Add Real Server

$ ipvsadm -a -t 192.168.1.100:80 -r 172.16.16.2:80 -m -w 1

Modify Scheduler

$ ipvsadm -E -t 192.168.1.100:80 -s wlc

Delete Service

$ ipvsadm -D -t 192.168.1.100:80

LVS‑NAT & Keepalived Deployment Example

Although LVS‑NAT has drawbacks, its versatility makes it common in commercial LB products (e.g., Cisco LocalDirector, F5 BIG‑IP).

TCP Load‑Balancing Scenario

IP Planning

Client (CIP): 192.168.1.100/24

DS1: external 192.168.1.110/24, internal 10.0.0.103/24

DS2: external 192.168.1.111/24, internal 10.0.0.104/24

Virtual Server VIP: 192.168.1.112

DS internal DIP: 10.0.0.105

RS1 RIP: 10.0.0.101, gateway 10.0.0.105

RS2 RIP: 10.0.0.102, gateway 10.0.0.105

Network Topology

DS1 Configuration

Start ipvsadm and keepalived:

$ touch /etc/sysconfig/ipvsadm
$ systemctl start ipvsadm.service
$ yum install -y keepalived ipvsadm
$ keepalived --version
$ systemctl start keepalived

Sample /etc/keepalived/keepalived.conf (master):

global_defs {
    lvs_id LVS_01
}

vrrp_sync_group VG1 {
    group {
        VI_1
        VI_GATEWAY
    }
}

vrrp_instance VI_1 {
    state MASTER
    interface eno16777736
    lvs_sync_daemon_inteface eno16777736
    virtual_router_id 51
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.112
    }
}

vrrp_instance VI_GATEWAY {
    state MASTER
    interface eno33554960
    lvs_sync_daemon_inteface eno33554960
    virtual_router_id 52
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass example
    }
    virtual_ipaddress {
        10.0.0.105
    }
}

virtual_server 192.168.1.112 80 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol TCP
    real_server 10.0.0.101 80 { weight 1 }
    real_server 10.0.0.102 80 { weight 1 }
}

Enable IP forwarding and clear firewall rules:

# echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.conf
# sysctl -p
# iptables -F -t filter
# iptables -F -t raw
# iptables -F -t mangle
# iptables -F -t nat

RS Configuration

Set default gateway to DIP and install a simple HTTP server:

# echo 'GATEWAY=10.0.0.105' >> /etc/sysconfig/network-scripts/ifcfg-eno33554960
# yum install -y httpd
# echo '<html><body><h1>RS1</h1></body></html>' > /var/www/html/index.html
# systemctl start httpd

Verification

From a client, repeatedly curl the VIP; responses alternate between RS1 and RS2, confirming round‑robin scheduling.

$ curl 192.168.1.112
<html><body><h1>RS2</h1></body></html>
$ curl 192.168.1.112
<html><body><h1>RS1</h1></body></html>

Check connection tables on master and backup routers using ipvsadm -Lnc. After a master failure, the VIP fails over to the backup, and connections continue.

UDP Load‑Balancing Scenario

DS Configuration

Add a UDP virtual server to the same keepalived configuration:

virtual_server 192.168.1.112 9999 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    nat_mask 255.255.255.0
    protocol UDP
    real_server 10.0.0.101 9999 { weight 1 }
    real_server 10.0.0.102 9999 { weight 1 }
}

RS Configuration

Install nc to listen on UDP port 9999 and write received data to a file:

# yum install -y nc
# nc -ul 9999 > file.txt

Verification

Send files from a client using netcat; each file is received by a different RS, confirming load‑balanced UDP forwarding.

$ nc -u -w 1 192.168.1.112 9999 < 1.txt
$ nc -u -w 1 192.168.1.112 9999 < 2.txt

Check the IPVS connection table to see entries for both real servers.

$ ipvsadm -Lnc
IPVS connection entries
pro expire state   source               virtual            destination
UDP 04:20 UDP      192.168.1.100:65136 192.168.1.112:9999 10.0.0.101:9999
UDP 04:19 UDP      192.168.1.100:51930 192.168.1.112:9999 10.0.0.102:9999