Build a Highly Available Kubernetes Cluster on QingCloud LB – Step‑by‑Step Guide

Introduction

The article explains how to build a highly available Kubernetes (k8s) cluster on QingCloud using its Load Balancer (LB) service, summarizing the problems encountered and solutions applied.

Cluster Architecture

A three‑master and two‑worker node layout is used, with each host running CentOS 7.6 and Docker as the container runtime.

Host Inventory

Host System   Role          IP          Hostname   Container Runtime
------------------------------------------------------------
CentOS7.6     Control Node  192.168.0.1 ks-m1      docker
CentOS7.6     Control Node  192.168.0.2 ks-m2      docker
CentOS7.6     Control Node  192.168.0.3 ks-m3      docker
CentOS7.6     Worker Node   192.168.0.4 ks-n1      docker
CentOS7.6     Worker Node   192.168.0.5 ks-n2      docker

Environment Preparation

1. Hostname Configuration

# On 192.168.0.1
hostnamectl set-hostname ks-m1 && bash
# On 192.168.0.2
hostnamectl set-hostname ks-m2 && bash
# On 192.168.0.3
hostnamectl set-hostname ks-m3 && bash
# On 192.168.0.4
hostnamectl set-hostname ks-n1 && bash
# On 192.168.0.5
hostnamectl set-hostname ks-n2 && bash

2. /etc/hosts Update

# Append the following lines to /etc/hosts on all nodes
192.168.0.1 ks-m1
192.168.0.2 ks-m2
192.168.0.3 ks-m3
192.168.0.4 ks-n1
192.168.0.5 ks-n2

3. Disable Firewall, SELinux and Swap

# Stop and disable firewalld
systemctl stop firewalld
systemctl disable firewalld
# Permanently disable SELinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
# Comment out swap entries and turn swap off
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a

4. Kernel Upgrade

# Import ELRepo public key
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# Install ELRepo repository
yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm -y
# List available kernels
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
# Install the latest long‑term kernel
yum --enablerepo=elrepo-kernel install kernel-lt -y
# Set the new kernel as default and rebuild grub config (BIOS example)
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
# Reboot and verify
reboot
uname -r

5. Time Synchronization

# Install ntpdate and sync time
yum install ntpdate -y
ntpdate cn.pool.ntp.org
# Add hourly sync to crontab
(crontab -l ; echo "* */1 * * * /usr/sbin/ntpdate cn.pool.ntp.org") | crontab -
service crond restart
# Or use chronyd
yum install -y chronyd
systemctl start chronyd && systemctl enable chronyd
sed -i 's/^#server .*/server ntp1.aliyun.com iburst/' /etc/chrony.conf

Docker Installation and Configuration

# Install Docker CE 20.10.8 (or a specific version)
yum install -y docker-ce-20.10.8 docker-ce-cli-20.10.8 containerd.io
systemctl start docker && systemctl enable docker && systemctl status docker
# Create custom daemon.json (example content)
cat > /etc/docker/daemon.json <<EOF
{"data-root":"/data/docker/lib","log-driver":"json-file","registry-mirrors":["https://***.mirror.aliyuncs.com"],"bip":"10.128.10.1/24","exec-opts":["native.cgroupdriver=systemd"],"insecure-registries":["harbor_ip:port"]}
EOF
systemctl restart docker

Kubernetes Packages

# Install kubelet, kubeadm and kubectl version 1.22.12
yum install -y kubelet-1.22.12 kubeadm-1.22.12 kubectl-1.22.12
systemctl enable kubelet && systemctl start kubelet

Master Node Initialization

# Create kubeadm config file (kubeadm-init.yaml) – example excerpt
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.0.1
  bindPort: 6443
nodeRegistration:
  name: ks-m1
  criSocket: /var/run/dockershim.sock
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: 172.17.0.252:6443
kubernetesVersion: 1.22.12
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.10.0.0/16
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

# Initialize the cluster
kubeadm init --config kubeadm-init.yaml --upload-certs

# Set up kubectl for the admin user
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

Adding Additional Masters and Workers

# Join a new master (replace token and hash as needed)
kubeadm join 172.17.0.252:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane

# Join a worker node
kubeadm join 172.17.0.252:6443 \
  --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

Network Plugin – Calico

# Download Calico manifest
curl -O https://projectcalico.docs.tigera.io/archive/v3.22/manifests/calico.yaml
# Edit calico.yaml – set interface detection (example)
# Add under spec.template.spec.containers[0].env:
# - name: IP_AUTODETECTION_METHOD
#   value: "interface=eth.*"
# Apply the manifest
kubectl apply -f calico.yaml

CoreDNS

CoreDNS is installed automatically with the Calico network plugin.

Metrics Server

# Download the latest components.yaml
curl -LO https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# Replace the image with an Alibaba Cloud mirror
sed -i 's|k8s.gcr.io/metrics-server/metrics-server:.*|registry.aliyuncs.com/google_containers/metrics-server:v0.6.1|' components.yaml
# Add insecure TLS flag to the args section
sed -i '/- --secure-port=4443/a\    - --kubelet-insecure-tls' components.yaml
# Apply the manifest
kubectl apply -f components.yaml

APIServer Aggregation

# Edit the static pod manifest for kube-apiserver (usually /etc/kubernetes/manifests/kube-apiserver.yaml)
# Add the flag:
- --enable-aggregator-routing=true
# The apiserver pod will restart automatically.

Verification

# Check node status
kubectl get nodes
# Verify pods in kube-system namespace
kubectl get pods -n kube-system
# View resource usage
kubectl top nodes

Load Balancer Configuration (QingCloud LB)

Create a QingCloud LB, assign an IP, add a TCP listener on port 6443, register the three master nodes, and ensure the security group permits traffic. The LB’s internal IP (e.g., 172.17.0.252) is used as the controlPlaneEndpoint in the kubeadm configuration.