Master Nginx, Keepalived, and LVS: Build a High‑Availability Load‑Balancing Cluster

1. Install Nginx

1.1 Download the appropriate package from nginx.org (stable version)

1.2 Upload the package to the Linux server

1.3 Install required dependencies

(1) Install GCC yum install gcc-c++ (2) Install PCRE (regular‑expression library) yum install -y pcre pcre-devel (3) Install zlib (compression library) yum install -y zlib zlib-devel (4) Install OpenSSL (HTTPS support)

yum install -y openssl openssl-devel

1.4 Extract the source archive

tar -zxvf nginx-1.16.1.tar.gz

1.5 Create a temporary directory for Nginx

mkdir -p /var/temp/nginx

1.6 Configure the source to generate a Makefile

./configure \
  --prefix=/usr/local/nginx \
  --pid-path=/var/run/nginx/nginx.pid \
  --lock-path=/var/lock/nginx.lock \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --with-http_gzip_static_module \
  --http-client-body-temp-path=/var/temp/nginx/client \
  --http-proxy-temp-path=/var/temp/nginx/proxy \
  --http-fastcgi-temp-path=/var/temp/nginx/fastcgi \
  --http-uwsgi-temp-path=/var/temp/nginx/uwsgi \
  --http-scgi-temp-path=/var/temp/nginx/scgi

Note: backslashes indicate line continuation for readability.

1.7 Compile and install

make
make install

1.8 Start/stop/reload Nginx

nginx               # start
./nginx -s stop     # stop
./nginx -s reload   # reload

2. Configure Reverse Proxy

Define an upstream group:

upstream [proxyName] {
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

Configure the server block:

server {
    listen 80;
    server_name www.tomcats.com;
    location / {
        proxy_pass http://[proxyName];
    }
}

3. Configure Load Balancing Methods

Default is round‑robin. Additional methods:

Weighted round‑robin:

upstream [proxyName] {
    server 192.168.1.173:8080 weight=1;
    server 192.168.1.174:8080 weight=5;
    server 192.168.1.175:8080 weight=2;
}

IP hash:

upstream [proxyName] {
    ip_hash;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

URL hash:

upstream [proxyName] {
    hash $request_url;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

Least connections:

upstream [proxyName] {
    least_conn;
    server 192.168.1.173:8080;
    server 192.168.1.174:8080;
    server 192.168.1.175:8080;
}

4. Upstream Directive Parameters

max_conns

: limit maximum simultaneous connections (commercial edition before 1.11.5) slow_start: seconds to ramp weight from 1 to configured value (not for hash or random LB, ineffective if only one server) down: mark server as unavailable backup: designate a backup server (used only when primary servers fail) max_fails: number of failures before marking server down (default 1) fail_timeout: time to wait before retrying a failed server (default 10 seconds)

5. Configure SSL for HTTPS

Install the SSL module (http_ssl_module) and re‑configure Nginx:

./configure \
  --prefix=/usr/local/nginx \
  ...
  --with-http_ssl_module

Compile and install again:

make
make install

Copy *.crt and *.key to /usr/local/nginx/conf and add an HTTPS server block:

server {
    listen 443;
    server_name www.imoocdsp.com;
    ssl on;
    ssl_certificate 1_www.imoocdsp.com_bundle.crt;
    ssl_certificate_key 2_www.imoocdsp.com.key;
    ssl_session_cache shared:SSL:1m;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE;
    ssl_prefer_server_ciphers on;
    location / {
        proxy_pass http://tomcats/;
        index index.html index.htm;
    }
}

6. High‑Availability with Keepalived

6.1 Install Keepalived

Download, extract, configure and install:

https://www.keepalived.org/download.html
tar -zxvf keepalived-2.0.18.tar.gz
./configure --prefix=/usr/local/keepalived --sysconf=/etc
make && make install

Install required libnl libraries if you see the IPv6 warning:

yum -y install libnl libnl-devel

6.2 Create configuration file /etc/keepalived/keepalived.conf

global_defs {
    router_id keep_171
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.161
    }
}

6.3 Register Keepalived as a system service

Copy etc/init.d/keepalived to /etc/init.d/ Copy etc/sysconfig/keepalived to /etc/sysconfig/ Reload systemd: systemctl daemon-reload Start/stop/restart with

systemctl start|stop|restart keepalived.service

6.4 Script to monitor Nginx and restart Keepalived if Nginx fails

#!/bin/bash
A=`ps -C nginx --no-header | wc -l`
if [ $A -eq 0 ]; then
    /usr/local/nginx/sbin/nginx
    sleep 3
    if [ `ps -C nginx --no-header | wc -l` -eq 0 ]; then
        killall keepalived
    fi
fi

Make it executable:

chmod +x /etc/keepalived/check_nginx_alive_or_not.sh

Add a track script in the Keepalived config:

vrrp_script check_nginx_alive {
    script "/etc/keepalived/check_nginx_alive_or_not.sh"
    interval 2
    weight 10
}

track_script { check_nginx_alive }

systemctl restart keepalived

7. LVS (Linux Virtual Server) High‑Availability Load Balancing

7.1 Why combine LVS with Nginx?

LVS works at layer‑4 and can handle higher throughput than Nginx's layer‑7 proxy.

Using LVS for traffic distribution and Nginx for application‑level processing yields better performance.

7.2 LVS Modes

NAT : All traffic (request and response) passes through the LVS node.

TUN : Requests go through LVS, responses are sent directly from the real server via a tunnel.

DR (Direct Routing): Requests are distributed by LVS, responses bypass LVS and are sent directly to the client.

7.3 Build LVS in DR mode

Disable NetworkManager to avoid interface conflicts:

systemctl stop NetworkManager
systemctl disable NetworkManager

Create a virtual IP on the LVS node (e.g., 192.168.1.150):

cp ifcfg-ens33 ifcfg-ens33:1
# edit the copy
DEVICE="ens33:1"
ONBOOT="yes"
IPADDR=192.168.1.150
NETMASK=255.255.255.0
BOOTPROTO=static
service network restart

Install ipvsadm (the LVS management tool): yum install ipvsadm Configure ARP settings in /etc/sysctl.conf:

# configuration for LVS
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
sysctl -p

Add a route for the virtual IP on the loopback device:

route add -host 192.168.1.150 dev lo:1
echo "route add -host 192.168.1.150 dev lo:1" >> /etc/rc.local

Create the LVS virtual service (port 80, DR mode, round‑robin): ipvsadm -A -t 192.168.1.150:80 -s rr -p 5 Add real servers:

ipvsadm -a -t 192.168.1.150:80 -r 192.168.1.171:80 -g
ipvsadm -a -t 192.168.1.150:80 -r 192.168.1.172:80 -g
ipvsadm -S   # save rules

Check the configuration:

ipvsadm -Ln          # list services
ipvsadm -Ln --stats  # show statistics

8. Build a Keepalived + LVS + Nginx HA Cluster

8.1 Clean previous ipvsadm rules

ipvsadm -C

8.2 Configure Keepalived on the master LVS node

global_defs {
    router_id keep_151
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 41
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress { 192.168.1.150 }
}

virtual_server 192.168.1.150 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 5
    protocol TCP
    real_server 192.168.1.171 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
    real_server 192.168.1.172 80 {
        weight 1
        TCP_CHECK {
            connect_port 80
            connect_timeout 2
            nb_get_retry 2
            delay_before_retry 3
        }
    }
}

systemctl restart keepalived

8.3 Configure Keepalived on the backup LVS node (state BACKUP, lower priority)

global_defs { router_id keep_152 }

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 41
    priority 50
    advert_int 1
    authentication { auth_type PASS auth_pass 1111 }
    virtual_ipaddress { 192.168.1.150 }
}

virtual_server 192.168.1.150 80 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    persistence_timeout 5
    protocol TCP
    real_server 192.168.1.171 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 2 nb_get_retry 2 delay_before_retry 3 } }
    real_server 192.168.1.172 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 2 nb_get_retry 2 delay_before_retry 3 } }
}

systemctl restart keepalived

8.4 Result

Clients access the virtual IP (192.168.1.150). The master node distributes traffic to the Nginx real servers via LVS DR mode. If the master fails, Keepalived promotes the backup, ensuring continuous service.