How to Build a Scalable E‑Commerce Site with LVS Load‑Balancing (NAT, DR, SSL)

Background

As traffic grew, the site’s QPS increased from 1,000 to 3,000, overwhelming a single LNMP server. To handle more users, a load‑balancing solution based on LVS was required.

Project Overview

Full architecture diagrams are available at the original links (removed). The following sections detail the practical implementation.

Preparation

1. Install ipvsadm yum install ipvsadm -y Key files:

Program: /usr/sbin/ipvsadm Save rules: /usr/sbin/ipvsadm-save Restore rules: /usr/sbin/ipvsadm-restore Config: /etc/sysconfig/ipvsadm-config 2. Verify kernel support for IPVS grep -i -C 10 "ipvs" /boot/config-$(uname -r) 3. Clear firewall and disable SELinux

iptables -F && setenforce 0

Practical Exercise 1 – LVS NAT Mode

Architecture diagram:

Environment

lvs‑server: VIP 172.17.1.6, DIP 192.168.30.106 (load balancer, routing enabled)

rs01: RIP 192.168.30.107 (backend)

rs02: RIP 192.168.30.7 (backend)

Configure LVS on the load balancer

yum -y install ipvsadm

# Create virtual service on port 80 with WRR scheduling
ipvsadm -A -t 172.17.1.6:80 -s wrr
# Add real servers in NAT mode with weight 1
ipvsadm -a -t 172.17.1.6:80 -r 192.168.30.107:80 -m -w 1
ipvsadm -a -t 172.17.1.6:80 -r 192.168.30.7:80 -m -w 1

Enable IP forwarding:

echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p

Configure backend servers

# Start services
systemctl start nginx
systemctl start php-mysql
systemctl start mariadb
# Set default gateway to the load balancer
route add default gw 192.168.30.106

Test from a Windows client by accessing http://172.17.1.6/ and verify connections with ipvsadm -L -n.

Practical Exercise 2 – LVS DR Mode

Architecture diagram:

Environment

lvs‑server: VIP 172.17.100.100 (load balancer)

rs01: RIP 172.17.1.7 (backend)

rs02: RIP 172.17.22.22 (backend)

Configure VIP on the load balancer

ifconfig eth0:0 172.17.100.100 netmask 255.255.255.255 broadcast 172.17.100.100 up
route -host 172.17.100.100 dev eth0:0

Set up virtual service and DR real servers:

# Virtual service on port 80, WRR scheduling
ipvsadm -A -t 172.17.100.100:80 -s wrr
# Add DR real servers
ipvsadm -a -t 172.17.100.100:80 -r 172.17.1.7:80 -g -w 1
ipvsadm -a -t 172.17.100.100:80 -r 172.17.22.22:80 -g -w 1

On each real server, bind the VIP to the loopback interface and adjust ARP settings to avoid duplicate replies:

# Bind VIP
ifconfig lo:0 172.17.100.100 netmask 255.255.255.255 broadcast 172.17.100.100 up
# ARP ignore/announce
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

Persist settings in /etc/sysctl.conf and reload.

Practical Exercise 3 – Persistent Connections for Ports 80 & 443

Architecture diagram:

Steps

Mark packets for ports 80 and 443 using iptables:

# Mark port 80
iptables -t mangle -A PREROUTING -d 172.17.100.100 -p tcp --dport 80 -j MARK --set-mark 99
# Mark port 443
iptables -t mangle -A PREROUTING -d 172.17.100.100 -p tcp --dport 443 -j MARK --set-mark 99

Create a virtual service based on the mark and enable persistence:

ipvsadm -A -f 99 -s rr -p
# Add backend servers in DR mode
ipvsadm -a -f 99 -r 172.17.1.7 -g
ipvsadm -a -f 99 -r 172.17.22.22 -g

The -p flag enables persistent connections.

Practical Exercise 4 – SSL Termination

1. Create certificate directory mkdir /etc/nginx/ssl 2. Generate self‑signed certificate and key

# Example commands (actual generation may vary)
cd /etc/pki/tls/certs/
make nginx.crt
openssl rsa -in nginx.key -out nginx.key   # decrypt private key

Copy them to the SSL directory: cp nginx.crt nginx.key /etc/nginx/ssl/ 3. Add HTTPS server block to Nginx

server {
    listen 443 ssl;
    server_name www.along.com;
    ssl on;
    ssl_certificate /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;
    ssl_session_cache shared:sslcache:20m;
    ssl_session_timeout 10m;
}

Test by accessing https://172.17.1.6/ from a browser and trust the self‑signed certificate.

Multiple HTTPS Virtual Hosts

Because Nginx supports multiple virtual hosts, you can create additional certificates (nginx2.crt, nginx3.crt) and corresponding server blocks, each pointing to a different FQDN.

After copying the certificates and keys to /etc/nginx/ssl/ and configuring the server blocks, restart Nginx: systemctl restart nginx All tests confirm that the LVS cluster can balance HTTP and HTTPS traffic, maintain session persistence, and scale the e‑commerce site to handle increased load.