Step-by-Step Guide to Deploy a Multi-Node Kubernetes Cluster on CentOS

1. Installation Requirements

Before starting, the machines for the Kubernetes cluster must meet the following conditions:

Three machines running CentOS 7.7 (minimal installation)

Hardware: 2 GB RAM, at least 2 vCPU, 30 GB+ disk

All machines can communicate with each other and have internet access

Use NAT network model as appropriate

2. Installation Planning

(1) Server Planning

Role

IP

master

192.168.50.128

node1

192.168.50.131

node2

192.168.50.132

(2) Software Environment

Software

Version

Docker

19.03.12

Kubernetes

1.18.6

Etcd

3.4.9

3. System Initialization

3.1 Configure Hostname

# hostnamectl set-hostname master
# hostnamectl set-hostname node1
# hostnamectl set-hostname node2

3.2 Update /etc/hosts

# cat >> /etc/hosts <<EOF
192.168.50.128 master
192.168.50.128 node0
192.168.50.131 node1
192.168.50.132 node2
EOF

3.3 Configure password‑less SSH

# ssh-keygen -t rsa -b 1200
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2

3.4 (Optional) Upgrade Kernel

# yum localinstall -y kernel-*.rpm
# grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
# grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
# echo "please reboot your system quickly!!!"

3.5 Disable firewall and SELinux

# systemctl disable --now firewalld
# setenforce 0
# sed -i 's/enforcing/disabled/' /etc/selinux/config

3.6 Disable swap

# swapoff -a
# sed -i.bak 's/^.*centos-swap/#&/' /etc/fstab

3.7 Optimize kernel parameters

# cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward=1
vm.swappiness=0
EOF
# sysctl --system

3.8 Set timezone and NTP

# ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
# yum install dnf -y
# dnf makecache
# dnf install ntpdate -y
# ntpdate ntp.aliyun.com

3.9 Configure YUM mirrors (Aliyun)

# curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
# curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

4. Deploy Etcd Cluster

4.1 Install Etcd binary

# tar xf etcd-v3.4.9-linux-amd64.tar.gz
# cp -a etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /usr/local/bin

4.2 Create Etcd configuration

Option

Value

ETCD_NAME

etcd-01 (or etcd-02, etcd-03)

ETCD_DATA_DIR

/var/lib/etcd/default.etcd

ETCD_LISTEN_PEER_URLS

https://

node_ip

:2380

ETCD_LISTEN_CLIENT_URLS

https://

node_ip

:2379,http://127.0.0.1:2379

ETCD_INITIAL_ADVERTISE_PEER_URLS

https://

node_ip

:2380

ETCD_ADVERTISE_CLIENT_URLS

https://

node_ip

:2379

ETCD_INITIAL_CLUSTER

etcd-01=https://192.168.50.128:2380,etcd-02=https://192.168.50.131:2380,etcd-03=https://192.168.50.132:2380

ETCD_INITIAL_CLUSTER_TOKEN

etcd-cluster

ETCD_INITIAL_CLUSTER_STATE

new

4.3 Create systemd service

# cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/cfg/etcd.conf
ExecStart=/usr/local/bin/etcd \
--cert-file=/etc/etcd/ssl/server.pem \
--key-file=/etc/etcd/ssl/server-key.pem \
--peer-cert-file=/etc/etcd/ssl/server.pem \
--peer-key-file=/etc/etcd/ssl/server-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
# systemctl daemon-reload
# systemctl enable --now etcd

4.4 Verify health

# etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/server.pem --key=/etc/etcd/ssl/server-key.pem --endpoints="https://192.168.50.128:2379,https://192.168.50.131:2379,https://192.168.50.132:2379" endpoint health

5. Deploy Docker

5.1 Add Docker CE repo (Aliyun)

# curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

5.2 Install Docker 18.09.9

# dnf install -y docker-ce-18.09.9 docker-ce-cli-18.09.9
# systemctl enable --now docker
# docker --version

5.3 Configure registry mirror

# cat > /etc/docker/daemon.json <<EOF
{
  "registry-mirrors": ["https://f1bhsuge.mirror.aliyuncs.com"]
}
EOF
# systemctl restart docker

6. Deploy Master Node Components

6.1 Create Kubernetes root CA

# mkdir -p ~/TLS/k8s && cd ~/TLS/k8s
# cat > ca-config.json <<EOF
{
  "signing": {
    "default": {"expiry":"87600h"},
    "profiles": {"kubernetes": {"expiry":"87600h","usages":["signing","key encipherment","server auth","client auth"]}}
  }
}
EOF
# cat > ca-csr.json <<EOF
{
  "CN":"kubernetes",
  "key":{"algo":"rsa","size":2048},
  "names":[{"C":"CN","L":"Beijing","ST":"Beijing","O":"k8s","OU":"System"}]
}
EOF
# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# ls ca*.pem

6.2 Generate kube-apiserver certificate

# cat > server-csr.json <<EOF
{
  "CN":"kubernetes",
  "hosts":["10.0.0.1","127.0.0.1","192.168.50.128","192.168.50.131","192.168.50.132","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],
  "key":{"algo":"rsa","size":2048},
  "names":[{"C":"CN","L":"BeiJing","ST":"BeiJing","O":"k8s","OU":"System"}]
}
EOF
# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
# cp server*.pem ca*.pem /etc/kubernetes/ssl

6.3 kube-apiserver configuration

# cat > /etc/kubernetes/cfg/kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/var/log/kubernetes \
--advertise-address=192.168.50.128 \
--default-not-ready-toleration-seconds=360 \
--default-unreachable-toleration-seconds=360 \
--max-mutating-requests-inflight=2000 \
--max-requests-inflight=4000 \
--default-watch-cache-size=200 \
--delete-collection-workers=2 \
--bind-address=192.168.50.128 \
--secure-port=6443 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--service-node-port-range=1024-32767 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/etc/kubernetes/ssl/token.csv \
--kubelet-client-certificate=/etc/kubernetes/ssl/server.pem \
--kubelet-client-key=/etc/kubernetes/ssl/server-key.pem \
--tls-cert-file=/etc/kubernetes/ssl/server.pem \
--tls-private-key-file=/etc/kubernetes/ssl/server-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kubernetes/k8s-audit.log \
--etcd-servers=https://192.168.50.128:2379,https://192.168.50.131:2379,https://192.168.50.132:2379 \
--etcd-cafile=/etc/etcd/ssl/ca.pem \
--etcd-certfile=/etc/etcd/ssl/server.pem \
--etcd-keyfile=/etc/etcd/ssl/server-key.pem"
EOF

6.4 Create token for kubelet bootstrapping

# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
# cat > /etc/kubernetes/ssl/token.csv <<EOF
<generated-token>,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

6.5 Systemd unit for kube-apiserver

# cat > /usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
# systemctl daemon-reload
# systemctl enable --now kube-apiserver

6.6 Authorize kubelet‑bootstrap user

# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

6.7 Deploy kube-controller-manager

# cat > /etc/kubernetes/cfg/kube-controller-manager.conf <<EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/var/log/kubernetes \
--leader-elect=false \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF

# cat > /usr/lib/systemd/system/kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
Type=notify

[Install]
WantedBy=multi-user.target
EOF

# systemctl daemon-reload
# systemctl enable --now kube-controller-manager

6.8 Deploy kube-scheduler

# cat > /etc/kubernetes/cfg/kube-scheduler.conf <<EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/var/log/kubernetes \
--leader-elect=false \
--master=http://127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--address=127.0.0.1"
EOF

# cat > /usr/lib/systemd/system/kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

# systemctl daemon-reload
# systemctl enable --now kube-scheduler

7. Deploy Worker Nodes

7.1 Copy binaries to each node

# scp /usr/local/bin/kubelet root@nodeX:/usr/local/bin
# scp /usr/local/bin/kube-proxy root@nodeX:/usr/local/bin

7.2 kubelet configuration

# cat > /etc/kubernetes/cfg/kubelet.conf <<EOF
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/var/log/kubernetes \
--hostname-override=nodeX \
--container-runtime=docker \
--network-plugin=cni \
--kubeconfig=/etc/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/etc/kubernetes/cfg/bootstrap.kubeconfig \
--config=/etc/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/etc/kubernetes/ssl \
--image-pull-progress-deadline=15m \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
EOF

7.3 kubelet-config.yml (partial)

# cat > /etc/kubernetes/cfg/kubelet-config.yml <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
    cacheTTL: 2m0s
  x509:
    clientCAFile: /etc/kubernetes/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

7.4 Generate bootstrap kubeconfig for the node

# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.50.128:6443 --kubeconfig=bootstrap.kubeconfig
# kubectl config set-credentials kubelet-bootstrap --token=<generated-token> --kubeconfig=bootstrap.kubeconfig
# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=bootstrap.kubeconfig
# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
# cp bootstrap.kubeconfig /etc/kubernetes/cfg/

7.5 Systemd unit for kubelet

# cat > /usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service

[Service]
EnvironmentFile=/etc/kubernetes/cfg/kubelet.conf
ExecStart=/usr/local/bin/kubelet $KUBELET_OPTS
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# systemctl daemon-reload
# systemctl enable --now kubelet

7.6 Install CNI plugins

# wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
# mkdir -p /opt/cni/bin
# tar xf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

7.7 Deploy kube-proxy

# cat > /etc/kubernetes/cfg/kube-proxy.conf <<EOF
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/var/log/kubernetes \
--config=/etc/kubernetes/cfg/kube-proxy-config.yml"
EOF

# cat > /etc/kubernetes/cfg/kube-proxy-config.yml <<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
healthzBindAddress: 0.0.0.0:10256
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  burst: 200
  kubeconfig: /etc/kubernetes/cfg/kube-proxy.kubeconfig
  qps: 100
hostnameOverride: nodeX
clusterCIDR: 10.0.0.0/24
EOF

# Generate kube-proxy certificate (cfssl) and kube-proxy.kubeconfig (similar to apiserver steps)

# cat > /usr/lib/systemd/system/kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/etc/kubernetes/cfg/kube-proxy.conf
ExecStart=/usr/local/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

# systemctl daemon-reload
# systemctl enable --now kube-proxy

7.8 Approve node certificate signing requests

# kubectl get csr
# kubectl certificate approve $(kubectl get csr -o name | grep Pending)
# kubectl get nodes

8. Install CNI network (Flannel)

# curl -o kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# sed -i 's/quay.io/quay-mirror.qiniu.com/g' kube-flannel.yml
# kubectl apply -f kube-flannel.yml

9. Deploy CoreDNS

# kubectl apply -f coredns.yaml   # modify Service clusterIP to 10.0.0.2 if needed
# kubectl get pods -n kube-system -l k8s-app=kube-dns

10. Test the cluster

10.1 Deploy nginx

# kubectl create deployment nginx --image=nginx
# kubectl expose deployment nginx --port=80 --type=NodePort
# kubectl get svc nginx -o wide

Access the service using the node IP and the allocated NodePort (for example, http://192.168.50.132:30249).

11. Install Kubernetes Dashboard

11.1 Deploy dashboard

# curl -o recommended.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml
# Edit the Service to type NodePort and set nodePort: 30001
# kubectl apply -f recommended.yaml

11.2 Create admin service account and bind cluster‑admin role

# kubectl create serviceaccount dashboard-admin -n kube-system
# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
# kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep dashboard-admin | awk '{print $1}') -o go-template='{{.data.token | base64decode}}'

Open https:// :30001 in a browser and log in with the retrieved token.