Mastering High-Concurrency Load Balancing with Linux LVS: A Step-by-Step Guide

Load Balancing Overview

With the rapid growth of mobile internet, large‑scale concurrent usage has become the norm for applications such as Weibo, Zhihu, and Toutiao. For Linux operations engineers, the ability to handle high‑concurrency scenarios is a key factor for high‑salary positions.

LVS Introduction

LVS (Linux Virtual Server) is a virtual server cluster system that provides cheap, transparent load‑balancing across multiple nodes. It was initiated in May 1998 and is one of the earliest free‑software projects in China.

Why Use LVS

LVS operates at Layer 7, allowing HTTP‑based routing strategies, but newer Nginx versions also support Layer 4 TCP load balancing. When concurrency exceeds Nginx limits, LVS becomes the preferred solution. Typical thresholds: 10 000 PV/day or under 10 000 concurrent requests can use Nginx; larger portals and e‑commerce sites require LVS.

LVS Cluster Setup

Cluster Environment

# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
# uname -a
Linux lb03 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Two web servers (web03, web04) will serve the backend, and a director server (lb03) will run LVS.

Installing Management Tools

yum -y install ipvsadm keepalived

Viewing LVS Status

# ipvsadm

Configuring the LVS Service (DR Mode)

Steps:

Bind a virtual IP (VIP) to eth0 on the director server.

Clear existing LVS rules: ipvsadm -C.

Set timeout values: ipvsadm --set 30 5 60.

Add a virtual service: ipvsadm -A -t 10.0.0.13:80 -s wrr -p 20.

Associate real servers: ipvsadm -a -t 10.0.0.13:80 -r 10.0.0.17:80 -g -w 1 and ipvsadm -a -t 10.0.0.13:80 -r 10.0.0.18:80 -g -w 1.

Verify configuration: ipvsadm -ln.

ARP Suppression on Real Servers

To avoid ARP conflicts, configure the following kernel parameters on each real server:

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

Apply with sysctl -p.

Testing the Cluster

Browser test: access http://10.0.0.13. Command‑line test: curl 10.0.0.13. Check ARP tables to ensure correct MAC bindings.

DR (Direct Routing) Mode Details

In DR mode the director rewrites the destination MAC address to the real server’s MAC while keeping the original source IP (CIP) and virtual IP (VIP). The real server replies directly to the client, bypassing the director for the response, which yields high efficiency.

NAT (Network Address Translation) Mode

Both request and response packets pass through the director. The director rewrites the destination IP to the real server (RIP) and later rewrites the source IP back to VIP before sending the response to the client.

TUN (Tunneling) Mode

Requests are encapsulated in an outer IP header (DIP → RIP) and sent through an IP tunnel. The real server decapsulates, processes the request, and replies directly to the client, avoiding the director for the response while still supporting cross‑VLAN deployments.

FULLNAT Mode

FULLNAT combines DNAT and SNAT, allowing real servers to reside in a different VLAN without needing the director’s IP as a gateway. It provides the flexibility of NAT with the performance benefits of DR.

LVS Scheduling Algorithms

RR – Round Robin

WRR – Weighted Round Robin

LC – Least Connections

WLC – Weighted Least Connections

LBLC – Locality‑Based Least Connections

LBLCR – Locality‑Based Least Connections with Replication

DH – Destination Hashing

SH – Source Hashing

Keepalived High‑Availability

Install Keepalived ( yum -y install keepalived) and configure /etc/keepalived/keepalived.conf to manage VIP failover and health checks for the LVS nodes.

# systemctl restart keepalived.service
# ipvsadm -ln   # verify virtual service
# ip a s eth0   # verify VIP presence

Troubleshooting Checklist

Common issues include ARP cache inconsistencies after VIP migration, mismatched VLANs, and kernel module availability. Use arping to broadcast updated ARP entries when a director fails over.

References

LVS official site: http://www.linuxvirtualserver.org/

Various Chinese tutorials covering LVS architecture, IP load‑balancing techniques, and scheduling.

Keepalived documentation for persistence timeout and health checks.