Master Nginx, Keepalived, and LVS: Build a High‑Availability Load‑Balancing Cluster

1. Nginx Installation

1. Download the appropriate Nginx package from the official website (http://nginx.org/), preferably the stable version

2. Upload the Nginx package to the Linux system

3. Install required dependencies

(1) Install gcc environment yum install gcc-c++ (2) Install PCRE library for regular expression parsing yum install -y pcre pcre-devel (3) Install zlib compression/decompression dependencies yum install -y zlib zlib-devel (4) Install OpenSSL for HTTPS support

yum install -y openssl openssl-devel

4. Extract the source package (the extracted files need to be compiled)

tar -zxvf nginx-1.16.1.tar.gz

5. Create a temporary directory for Nginx (required to avoid startup errors)

mkdir /var/temp/nginx -p

6. Configure the source to generate a Makefile

./configure \
    --prefix=/usr/local/nginx \
    --pid-path=/var/run/nginx/nginx.pid \
    --lock-path=/var/lock/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_gzip_static_module \
    --http-client-body-temp-path=/var/temp/nginx/client \
    --http-proxy-temp-path=/var/temp/nginx/proxy \
    --http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
    --http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
    --http-scgi-temp-path=/var/temp/nginx/scgi

Note: the backslashes indicate line continuation for readability.

7. Compile and install

make
make install

8. Start Nginx from the sbin directory

nginx          # start
./nginx -s stop   # stop
./nginx -s reload # reload

2. Configure Reverse Proxy

1. Define an upstream block

upstream [proxyName] {
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

2. Configure the server block

server {
    listen 80;
    server_name www.tomcats.com;

    location / {
        proxy_pass http://tomcats;
    }
}

3. Load‑Balancing Configuration

Nginx uses round‑robin by default.

1. Weighted round‑robin

upstream [proxyName] {
    server 192.168.1.173:8080 weight=1;
    server 192.168.1.174:8080 weight=5;
    server 192.168.1.175:8080 weight=2;
}

2. IP hash load‑balancing

upstream [proxyName] {
    ip_hash;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

3. URL hash load‑balancing

upstream [proxyName] {
    hash $request_url;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

4. Least connections load‑balancing

upstream [proxyName] {
    least_conn;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

4. Upstream Directive Parameters

max_conns

: limit maximum simultaneous connections (commercial edition before 1.11.5) slow_start: weight ramps up over the specified seconds (not for hash or random load‑balancing; ineffective if only one server; commercial edition only) down: mark server as unavailable backup: designate a backup server (used only when primary servers are down; not applicable to hash or random methods) max_fails: number of failed attempts before marking a server down (default 1) fail_timeout: time window for failure counting (default 10 seconds)

5. Keepalived Configuration for High Availability

1. Install Keepalived

https://www.keepalived.org/download.html

tar -zxvf keepalived-2.0.18.tar.gz

./configure --prefix=/usr/local/keepalived --sysconf=/etc

2. Install dependencies for IPVS with IPv6 support

yum -y install libnl libnl-devel

make && make install

3. Edit /etc/keepalived/keepalived.conf (example)

global_defs {
    router_id keep_171
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.161
    }
}

4. Register Keepalived as a system service (copy init scripts, reload systemd, start/stop/restart as needed).

6. Keepalived Script to Monitor Nginx

#!/bin/bash
A=`ps -C nginx --no-header | wc -l`
if [ $A -eq 0 ]; then
    /usr/local/nginx/sbin/nginx
    sleep 3
    if [ `ps -C nginx --no-header | wc -l` -eq 0 ]; then
        killall keepalived
    fi
fi

chmod +x /etc/keepalived/check_nginx_alive_or_not.sh

vrrp_script check_nginx_alive {
    script "/etc/keepalived/check_nginx_alive_or_not.sh"
    interval 2
    weight 10
}

track_script {
    check_nginx_alive
}

7. SSL Configuration for HTTPS

Install the SSL module (http_ssl_module) and re‑configure Nginx with --with-http_ssl_module:

./configure \
    --prefix=/usr/local/nginx \
    --pid-path=/var/run/nginx/nginx.pid \
    --lock-path=/var/lock/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_gzip_static_module \
    --http-client-body-temp-path=/var/temp/nginx/client \
    --http-proxy-temp-path=/var/temp/nginx/proxy \
    --http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
    --http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
    --http-scgi-temp-path=/var/temp/nginx/scgi \
    --with-http_ssl_module

make
make install

Add an HTTPS server block:

server {
    listen 443;
    server_name www.imoocdsp.com;
    ssl on;
    ssl_certificate 1_www.imoocdsp.com_bundle.crt;
    ssl_certificate_key 2_www.imoocdsp.com.key;
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE;
    ssl_prefer_server_ciphers on;
    location / {
        proxy_pass http://tomcats/;
        index index.html index.htm;
    }
}

8. LVS (Linux Virtual Server) High‑Availability Load‑Balancing

1. Why combine LVS with Nginx

LVS operates at layer‑4 and offers higher throughput than Nginx's layer‑7 balancing.

Layer‑7 processing is still needed for application‑level logic, so Nginx handles the content while LVS distributes connections.

LVS can forward requests without responding, reducing load on the balancer.

2. LVS Modes

(1) NAT mode – all traffic passes through LVS, suitable for up to ~10‑20 nodes.

(2) TUN mode – creates a tunnel; responses bypass LVS, but requires each real server to bind the virtual IP.

(3) DR mode – Direct Routing; responses go directly from the real server to the client, offering the best performance while keeping the virtual IP hidden.

3. Build LVS‑DR Mode

Disable NetworkManager to avoid interface conflicts:

systemctl stop NetworkManager
systemctl disable NetworkManager

Create a sub‑interface for the virtual IP on the LVS node:

cp ifcfg-ens33 ifcfg-ens33:1
# edit the file to set:
DEVICE="ens33:1"
ONBOOT="yes"
IPADDR=192.168.1.150
NETMASK=255.255.255.0
BOOTPROTO=static

service network restart

Configure ARP settings in /etc/sysctl.conf:

# configuration for LVS
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1

net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

sysctl -p

Add a host route for the virtual IP: route add -host 192.168.1.150 dev lo:1 Persist the route:

echo "route add -host 192.168.1.150 dev lo:1" >> /etc/rc.local

Install ipvsadm and create the LVS cluster:

yum install ipvsadm
ipvsadm -A -t 192.168.1.150:80 -s rr -p 5   # add virtual service
ipvsadm -a -t 192.168.1.150:80 -r 192.168.1.171:80 -g   # add real server 1
ipvsadm -a -t 192.168.1.150:80 -r 192.168.1.172:80 -g   # add real server 2
ipvsadm -S   # save rules

Check the configuration:

ipvsadm -Ln          # list services
ipvsadm -Ln --stats # show statistics

4. LVS Load‑Balancing Algorithms

Static algorithms: round‑robin (rr), weighted round‑robin (wrr), source hash (sh), destination hash (dh).

Dynamic algorithms: least connections (lc), weighted least connections (wlc), shortest expected delay (sed), never queue (nq).

9. Combine Keepalived, LVS, and Nginx for a High‑Availability Cluster

Clear any existing ipvsadm configuration before starting: ipvsadm -C Configure Keepalived on the master LVS node:

global_defs {
    router_id keep_151
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 41
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.150
    }
}

virtual_server 192.168.1.150 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 5
    protocol TCP
    real_server 192.168.1.171 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
    real_server 192.168.1.172 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
}

Configure Keepalived on the backup LVS node (change state to BACKUP and lower priority).

global_defs {
    router_id keep_152
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 41
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.150
    }
}

virtual_server 192.168.1.150 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 5
    protocol TCP
    real_server 192.168.1.171 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
    real_server 192.168.1.172 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
}

Start or restart Keepalived on both nodes:

systemctl restart keepalived

Note: Alibaba Cloud does not support manual network‑card configuration; a dedicated load‑balancing service must be purchased. Tencent Cloud supports manual configuration but limits the number of virtual IPs per NIC.