How to Build a Highly Available Kubernetes Cluster on CentOS with HAProxy and Keepalived

1. Installation Requirements

Before starting, the five machines that will form the Kubernetes cluster must meet the following conditions:

Five machines running CentOS 7.5+ (minimal installation)

Hardware: 2 GB RAM, at least 2 vCPU, 30 GB+ disk

All machines must have network connectivity between each other and to the Internet

2. Installation Steps

2.1 Pre‑processing

(1) Set Hostnames

Run the following commands on each node:

# hostnamectl set-hostname master1
# hostnamectl set-hostname master2
# hostnamectl set-hostname master3
# hostnamectl set-hostname node1
# hostnamectl set-hostname node2

(2) Update /etc/hosts

Add the IP‑hostname mappings for all nodes:

# cat >> /etc/hosts <<EOF
192.168.50.100 k8s-lb
192.168.50.128 master1
192.168.50.129 master2
192.168.50.130 master3
192.168.50.131 node1
192.168.50.132 node2
EOF

(3) Configure Password‑less SSH

Generate an RSA key pair on master1 and distribute the public key to the other nodes:

# ssh-keygen -t rsa -b 1200
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master1
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master2
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master3
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2

(4) Upgrade Kernel

Install a newer kernel from the ELRepo repository and reboot:

#!/bin/bash
# upgrade kernel by [email protected]
yum localinstall -y kernel-lt*
if [ $? -eq 0 ]; then
  grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
  grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
fi
echo "please reboot your system quick!!!"

Important: Reboot the machine after the upgrade.

(5) Disable Firewall and SELinux

# systemctl disable --now firewalld
# setenforce 0
# sed -i 's/enforcing/disabled/' /etc/selinux/config

(6) Disable Swap

# swapoff -a
# sed -i.bak 's/^.*centos-swap/#&/g' /etc/fstab

(7) Optimize Kernel Parameters

# cat > /etc/sysctl.d/k8s.conf <<EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.core.somaxconn = 16384
EOF
# sysctl --system

(8) Configure YUM Repositories

Replace the default CentOS repos with Alibaba Cloud mirrors:

# mv /etc/yum.repos.d/* /tmp
# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

(9) Set Timezone and NTP

# ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# yum install dnf ntpdate -y
# ntpdate ntp.aliyun.com

2.2 Install Docker

(1) Add Docker YUM repo

# curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

(2) Install Docker CE

# dnf install -y docker-ce docker-ce-cli
# systemctl enable --now docker
# docker --version

(3) Configure Docker Registry Mirror

# cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://f1bhsuge.mirror.aliyuncs.com"]
}
EOF
# systemctl restart docker

2.3 Install Kubernetes

(1) Add Kubernetes YUM repo

# cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
# dnf clean all && dnf makecache

(2) Install kubeadm, kubelet, kubectl (v1.18.6)

# dnf install -y kubelet-1.18.6 kubeadm-1.18.6 kubectl-1.18.6
# systemctl enable kubelet

2.4 Configure HAProxy + Keepalived for VIP

(1) Install keepalived and haproxy on the three master nodes

# dnf install -y keepalived haproxy

(2) HAProxy configuration (common for all masters)

global
  maxconn 2000
  ulimit-n 16384
  log 127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode http
  option httplog
  timeout connect 5000
  timeout client 50000
  timeout server 50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind *:8006
  mode http
  stats enable
  stats hide-version
  stats uri /stats
  stats refresh 30s
  stats realm "Haproxy Statistics"
  stats auth admin:admin

frontend k8s-master
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  server master1 192.168.50.128:6443 check inter 2000
  server master2 192.168.50.129:6443 check inter 2000
  server master3 192.168.50.130:6443 check inter 2000

(3) Keepalived configuration (example for master1)

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}

vrrp_script chk_kubernetes {
    script "/etc/keepalived/check_kubernetes.sh"
    interval 2
    weight -5
    fall 3
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    mcast_src_ip 192.168.50.128
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.50.100
    }
    #track_script { chk_kubernetes }
}

(4) Health‑check script

#!/bin/bash
function check_kubernetes() {
  for ((i=0;i<5;i++)); do
    apiserver_pid_id=$(pgrep kube-apiserver)
    if [[ ! -z $apiserver_pid_id ]]; then
        return
    else
        sleep 2
    fi
    apiserver_pid_id=0
  done
}
# 1:running 0:stopped
check_kubernetes
if [[ $apiserver_pid_id -eq 0 ]]; then
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

(5) Start services

# systemctl enable --now keepalived haproxy
# systemctl status keepalived haproxy

2.5 Deploy Master Nodes

(1) Generate kubeadm init config

# kubeadm config print init-defaults > kubeadm-init.yaml

Edit kubeadm-init.yaml to set the VIP (192.168.50.100) as advertiseAddress, add the VIP to certSANs, set controlPlaneEndpoint to 192.168.50.100:8443, and specify the pod network CIDR 10.244.0.0/16.

(2) Pull required images in advance

# kubeadm config images pull --config kubeadm-init.yaml

(3) Initialize the first master

# kubeadm init --config kubeadm-init.yaml --upload-certs

After successful init, copy /etc/kubernetes/admin.conf to $HOME/.kube/config and set KUBECONFIG environment variable.

(4) Join the remaining masters

# kubeadm join 192.168.50.100:8443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash> \
    --control-plane --certificate-key <key>

(5) Join worker nodes

# kubeadm join 192.168.50.100:8443 --token <token> \
    --discovery-token-ca-cert-hash sha256:<hash>

2.8 Install Network Plugin (Flannel)

Because the default GitHub URL is blocked in China, modify /etc/hosts to resolve raw.githubusercontent.com to 199.232.28.133, download the manifest, replace quay.io with quay-mirror.qiniu.com, and apply:

# curl -o kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# sed -i 's/quay.io/quay-mirror.qiniu.com/g' kube-flannel.yml
# kubectl apply -f kube-flannel.yml

Verify that all flannel pods are Running and that kubectl get nodes shows Ready status.

3 Test the Cluster

(1) Deploy an Nginx deployment

# kubectl create deployment nginx --image=nginx
# kubectl expose deployment nginx --port=80 --type=NodePort

Find the NodePort (e.g., 30249) and access http://<node_ip>:30249 from a browser.

(2) Install the Kubernetes Dashboard

# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
# edit the Service to type NodePort and set nodePort: 30001
# kubectl apply -f recommended.yaml

Retrieve the admin token:

# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

Access the dashboard at https://<node_ip>:30001 using the token, or create a cluster-admin binding for full privileges.

For any errors such as "error syncing endpoints with etc: context deadline exceeded", verify the Keepalived configuration, VIP, and priority settings.