Mastering Kubernetes Node Lifecycle: Init, Join, Maintain & Safe Decommission

Node Initialization: Prerequisites

Before a new node joins the cluster, system-level configuration must be completed to avoid compatibility issues.

1. Base Environment Setup (All Nodes)

# 1. Disable SELinux (K8s does not recommend enabling)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

# 2. Disable swap (K8s requires swap off to prevent memory scheduling anomalies)
swapoff -a
sed -i '/swap/s/^/#/' /etc/fstab

# 3. Configure kernel parameters (enable bridging, IP forwarding)
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system  # apply configuration

# 4. Install container runtime (containerd example, recommended for K8s 1.24+)
yum install -y containerd.io  # CentOS; Ubuntu uses apt install containerd.io
containerd config default > /etc/containerd/config.toml
# Replace sandbox image (default is foreign registry, switch to Alibaba Cloud mirror)
sed -i 's#k8s.gcr.io/pause#registry.aliyuncs.com/google_containers/pause#g' /etc/containerd/config.toml
systemctl restart containerd && systemctl enable containerd

2. Install K8s Components (All Nodes)

# 1. Add K8s yum repository (Alibaba Cloud for faster domestic access)
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyuncs.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyuncs.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyuncs.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 2. Install kubeadm, kubelet, kubectl (pin version to avoid drift, example uses 1.26.0)
yum install -y kubelet-1.26.0 kubeadm-1.26.0 kubectl-1.26.0
systemctl enable kubelet  # enable on boot, do not start yet (starts after joining cluster)

Joining the Cluster: Master and Worker Operations

Cluster initialization requires a master node first, then worker nodes join. Assumes an existing master or a new master created with kubeadm init.

1. Master Node Initialization (First Time Only)

# Initialize master, specify pod CIDR (avoid conflict with host network), APIServer address (master IP)
kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=192.168.1.100  # replace with master IP
  --image-repository=registry.aliyuncs.com/google_containers \  # Alibaba Cloud image repo
  --kubernetes-version=1.26.0

# Configure kubectl permissions (on master, otherwise kubectl cannot work)
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# Install network plugin (Calico example, mandatory for pod communication)
kubectl apply -f https://docs.projectcalico.org/v3.25/manifests/calico.yaml

2. Worker Node Joining

# 1. On master, generate join command (copy full output)
kubeadm token create --print-join-command
# Output example: kubeadm join 192.168.1.100:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx

# 2. On worker, execute the join command
kubeadm join 192.168.1.100:6443 --token xxx --discovery-token-ca-cert-hash sha256:xxx

# 3. Verify join success (on master)
kubectl get nodes
# Worker node status Ready indicates success

Daily Maintenance: Node Status and Attributes

After joining, monitor status and adjust labels/taints to match scheduling requirements.

1. Node Status Checks

# 1. View all node statuses (Ready is healthy)
kubectl get nodes -o wide

# 2. Inspect single node details (resource usage, pod distribution, events)
kubectl describe node worker-01  # replace with node name

# 3. Check node component logs (e.g., kubelet issues)
journalctl -u kubelet -f  # follow kubelet logs in real time

2. Label and Taint Management

Labels enable scheduling selection (e.g., target pods to specific nodes); taints repel scheduling (e.g., keep regular pods off master).

# 1. Add label to node (example: mark as database node)
kubectl label nodes worker-01 app=database

# 2. View node labels
kubectl get nodes worker-01 --show-labels

# 3. Add taint to node (example: master taint to prevent regular pod scheduling)
kubectl taint nodes master-01 node-role.kubernetes.io/master:NoSchedule

# 4. Remove taint (e.g., worker needs to accept pods again)
kubectl taint nodes worker-01 key:NoSchedule-

Safe Decommissioning: Prevent Data Loss and Cluster Issues

When a node must be taken offline (hardware replacement, retirement), never power off directly . First evict pods and clean resources.

1. Pre-Decommission: Evict Pods and Mark Unschedulable

# 1. Mark node as unschedulable (prevent new pods from being scheduled)
kubectl cordon worker-01

# 2. Evict all pods on node (--ignore-daemonsets: skip DaemonSet pods, they must stay)
kubectl drain worker-01 --ignore-daemonsets --delete-emptydir-data
# Note: --delete-emptydir-data removes EmptyDir data; omit if data must be preserved (manual handling required)

2. Remove Node from Cluster

# 1. On master, delete node (cluster-level removal)
kubectl delete node worker-01

# 2. On the node being decommissioned, clean K8s components (avoid residue affecting future rejoin)
kubeadm reset  # cleans kubelet, etcd data, network rules, etc.
rm -rf /etc/cni/net.d  # clean network plugin config
systemctl restart containerd  # restart container runtime

3. Verify Decommission Result

# On master, check node list to confirm decommissioned node is gone
kubectl get nodes

Key Considerations

Initialization Config : Kernel parameters, swap, and container runtime must be correct; otherwise node stays in NotReady.

Cluster Join : Token expires in 24 hours; if expired, regenerate on master with kubeadm token create --print-join-command.

Data Safety : When evicting pods, ensure StatefulSet pods have backed up data (e.g., PVC) to avoid loss.

Rejoin : Before rejoining, must run kubeadm reset to clear residue; otherwise certificate and config conflicts occur.

Following this workflow achieves stable Kubernetes node management from birth to retirement, avoiding cluster failures caused by improper operations.