How to Add a Master Node and Achieve High Availability in a Kubernetes Cluster

Kubernetes has become the most disruptive container orchestration technology in recent years and is widely used in production environments. Compared with earlier Docker‑Swarm solutions, Kubernetes offers a higher‑level management approach that simplifies project portability and architectural scaling.

In production, high availability is critical; unlike single‑master test setups, a production cluster should have at least two master nodes and two worker nodes so that if a master fails, the remaining master can continue serving kubelet requests via the API server.

Example cluster layout:

k8s-master1 192.168.175.128 k8s-master2 192.168.175.148 (new) k8s-node1 192.168.175.130 k8s-node2 192.168.175.131

1. High Availability Principle

Add a new master node and install nginx on each worker node. Nginx uses internal load balancing to proxy requests for the API server to both master nodes, ensuring master‑level HA. The scheduler and controller‑manager achieve HA through the leader-elect parameter.

Kubernetes management services (kube‑scheduler and kube‑controller‑manager) run a leader‑election algorithm backed by etcd. When leader-elect is enabled, each service attempts to become the leader, writes its endpoint to etcd, and periodically updates it. Other instances monitor the endpoint and take over if the leader stops updating, guaranteeing a single active leader even under high concurrency.

2. Initialization

On the new master host k8s-master2:

# iptables -F
# setenforce 0
# mkdir -pv /opt/kubernetes/{ssl,cfg,bin}

After adding the master, update the certificate request to include the new IP address, then distribute the new certificates to all nodes and restart the cluster components (k8s-master1, k8s-node1, k8s-node2).

3. Configure Master Components

Place the following configuration files under /opt/kubernetes/cfg/ to match the existing master1 settings.

KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.175.128:2379,https://192.168.175.130:2379,https://192.168.175.131:2379 \
--insecure-bind-address=127.0.0.1 \
--bind-address=192.168.175.148 \
--insecure-port=8080 \
--secure-port=6443 \
--advertise-address=192.168.175.148 \
--allow-privileged=true \
--service-cluster-ip-range=10.10.10.0/24 \
--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node \
--kubelet-https=true \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/server.pem \
--etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.10.10.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem"

KUBE_SCHEDULER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"

Start the three services, beginning with kube-apiserver; the other two can start in any order. At this stage the worker nodes are not yet reachable.

4. Install and Configure Nginx

Install nginx from the official repository:

# cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
EOF
# yum install nginx -y

Configure nginx for layer‑4 load balancing, listening on 127.0.0.1:6443, and forward traffic to the two master nodes. Update each worker node’s kubelet kubeconfig to point to 127.0.0.1:6443 so that requests are routed through nginx.

5. Cluster Testing

Verify that k8s-master2 can be accessed normally. Then manually shut down k8s-master1 and confirm that the cluster remains reachable, demonstrating successful high‑availability configuration.

Access is normal – the high‑availability cluster setup is successful!

Source: http://www.uml.org.cn/yunjisuan/202002053.asp