How to Build a 4‑Layer Reverse Proxy Cluster with Nginx and keepalived

Four‑Layer Reverse Proxy Cluster Planning

mfyxw10.mfyxw.com – 4‑layer load balancer (master) – 192.168.80.10

mfyxw20.mfyxw.com – 4‑layer load balancer (backup) – 192.168.80.20

Two Modes in keepalived

(1) master->backup mode
Once the master fails, the virtual IP automatically moves to the backup; when the master recovers, keepalived will reclaim the virtual IP even if nopreempt is set.
(2) backup->backup mode
When the master fails, the virtual IP moves to the backup, but after the original master recovers it will not preempt the IP, even with higher priority; the repaired master is typically used as the new backup.

1. Install Nginx for Reverse Proxy

# On mfyxw10 host
yum -y install nginx

# On mfyxw20 host
yum -y install nginx

2. Provide Reverse Proxy Configuration

# Append to /etc/nginx/nginx.conf on both hosts
stream {
    upstream kube-apiserver {
        server 192.168.80.30:6443 max_fails=3 fail_timeout=30s;
        server 192.168.80.40:6443 max_fails=3 fail_timeout=30s;
    }
    server {
        listen 7443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
    }
}

3. Verify Configuration and Start Nginx

# Test configuration and enable service on both hosts
nginx -t
systemctl enable --now nginx
systemctl status nginx

4. Install keepalived for High Availability

# Install on both hosts
yum -y install keepalived

5. keepalived Monitoring Script

# /etc/keepalived/check_port.sh
#!/bin/bash
CHK_PORT=$1
if [ -n "$CHK_PORT" ]; then
    PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
    if [ $PORT_PROCESS -eq 0 ]; then
        echo "Port $CHK_PORT Is Not Used, End."
        exit 1
    fi
else
    echo "Check Port Cant Be Empty!"
    exit 1
fi

6. keepalived Configuration for Master

# /etc/keepalived/keepalived.conf on mfyxw10
global_defs {
    router_id 192.168.80.10
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 192.168.80.10
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script { chk_nginx }
    virtual_ipaddress { 192.168.80.100 }
}

7. keepalived Configuration for Backup

# /etc/keepalived/keepalived.conf on mfyxw20
global_defs {
    router_id 192.168.80.20
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 7443"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 251
    priority 90
    advert_int 1
    mcast_src_ip 192.168.80.20
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script { chk_nginx }
    virtual_ipaddress { 192.168.80.100 }
}

8. Start keepalived and Verify

# On each host
systemctl enable --now keepalived
nginx -s reload
netstat -luntp | grep 7443
ip addr

9. Simulate Network Failure

# Stop keepalived on master, check VIP moves to backup
systemctl stop keepalived
ip addr   # on both hosts
# Restart keepalived on master, verify VIP returns
systemctl start keepalived
ip addr

10. Common keepalived Issues

systemctl cannot fully stop keepalived

When using yum to install keepalived (version keepalived‑1.3.5‑16.el7.x86_64), systemctl stop keepalived may fail, leaving the process running and causing “Can’t open PID file /var/run/keepalived.pid (yet?) after start”.

# Edit service file
vi /lib/systemd/system/keepalived.service
# Comment out or remove the line:
# KillMode=process