Mastering Kubernetes Node Isolation, Scaling, and Rolling Updates – Practical Commands and Tips

1. Node Isolation and Recovery

To prevent a node from receiving new Pods, edit its spec.unschedulable field to true and apply the change with kubectl replace -f unschedule_node.yaml. After replacement, kubectl get nodes shows the node status as Ready, SchedulingDisabled. Existing Pods on the node are not stopped automatically; they must be terminated manually. The same effect can be achieved without a YAML file by running

kubectl patch node <node-name> -p '{"spec":{"unschedulable":true}}'

. To re‑enable scheduling, set unschedulable back to false and repeat the replace or patch command.

2. Node Expansion

When a cluster runs out of capacity, add a new server, install Docker, Kubelet, and kube‑proxy, and configure the Kubelet to point to the existing Master URL. The node registers automatically via Kubelet’s auto‑registration mechanism. The following image illustrates the automatic registration process:

After registration, the Master includes the new node in its scheduling pool, allowing Pods to be scheduled onto it.

3. Pod Dynamic Scaling and Shrinking

Use kubectl scale rc <rc-name> --replicas=<number> to adjust the number of Pod replicas. For example, scaling the redis‑slave ReplicationController from 2 to 3 replicas:

$ kubectl scale rc redis-slave --replicas=3
scaled

Verify the new replica count with kubectl get pods. To shrink, set a lower replica count; the system terminates excess Pods accordingly.

4. Updating Resource Labels

Labels can be added, modified, or removed on existing objects using kubectl label. To add a label role=backend to a Pod:

$ kubectl label pod redis-master-bobr0 role=backend

View the updated Labels with kubectl get pods -Lrole. To delete a label, append a hyphen to the key: kubectl label pod redis-master-bobr0 role-. To change a label’s value, use the --overwrite flag: kubectl label pod redis-master-bobr0 role=master --overwrite.

5. Scheduling Pods to a Specific Node

Assign a custom label to the target node, e.g., zone=north:

$ kubectl label nodes kubernetes-minion1 zone=north

Then add a nodeSelector to the Pod’s specification matching that label:

apiVersion: v1
kind: ReplicationController
metadata:
  name: redis-master
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: master
        image: kubeguide/redis-master
        ports:
        - containerPort: 6379
      nodeSelector:
        zone: north

Create the Pod with kubectl create -f; the scheduler places it on the node bearing the zone=north label. Verify placement with kubectl get pods -o wide.

6. Rolling Updates

To upgrade a service without downtime, use kubectl rolling-update. Create a new ReplicationController definition (e.g., redis-master-v2) that includes a distinct version label:

apiVersion: v1
kind: ReplicationController
metadata:
  name: redis-master-v2
  labels:
    name: redis-master
    version: v2
spec:
  replicas: 1
  selector:
    name: redis-master
    version: v2
  template:
    metadata:
      labels:
        name: redis-master
        version: v2
    spec:
      containers:
      - name: master
        image: kubeguide/redis-master:2.0
        ports:
        - containerPort: 6379

Run the rolling update:

$ kubectl rolling-update redis-master -f redis-master-controller-v2.yaml

Kubectl creates the new RC, gradually reduces the old RC’s replicas to zero, and increases the new RC’s replicas until the target count is reached. The command logs each loop iteration, showing replica numbers for both RCs. After completion, the old RC is deleted and the new RC assumes the original name. A rollback can be performed with kubectl rolling-update … --rollback if needed.

7. Kubernetes Cluster High Availability

7.1 etcd High Availability

Deploy an etcd cluster of at least three nodes. Each node’s /etc/etcd/etcd.conf defines its name, data directory, peer URLs, client URLs, and cluster membership. The first node uses ETCD_INITIAL_CLUSTER_STATE="new"; subsequent nodes use "exist". Example configuration for the first node:

# [member]
ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd/etcd1"
ETCD_LISTEN_PEER_URLS="http://10.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.1:2379"
# [cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.1:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.1:2380,etcd2=http://10.0.0.2:2380,etcd3=http://10.0.0.3:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.1:2379"

Start the service with systemctl restart etcd. Add the remaining nodes with ETCD_INITIAL_CLUSTER_STATE="exist". Verify the cluster health:

$ etcdctl cluster-health
cluster is healthy
member ... is healthy

List members with etcdctl member list. Once healthy, configure the Kubernetes API server to use the etcd endpoints, e.g.,

--etcd-servers=http://10.0.0.1:4001,http://10.0.0.2:4001,http://10.0.0.3:4001

.

7.2 Master Component High Availability

Deploy the three Master services (kube‑apiserver, kube‑controller‑manager, kube‑scheduler) on at least three machines in an active‑standby‑standby configuration. Worker nodes access the Master cluster through a virtual IP or load‑balancer (e.g., Pacemaker). The diagram below shows a typical HA architecture: