How to Diagnose Kubernetes Node NotReady Issues: A Complete Step‑by‑Step Troubleshooting Guide

Problem Background

In production clusters a node may suddenly become NotReady, causing pods to stay Pending or be evicted. NotReady is a symptom; underlying causes include kubelet crashes, container‑runtime failures, network outages, etc. Blindly rebooting the node can destroy valuable evidence.

Applicable Scenarios

Node stays NotReady for more than 5 minutes

Business pods are evicted and replica count drops, triggering alerts

New nodes never become Ready after joining the cluster

Node becomes NotReady spontaneously (no upgrade or maintenance)

After a Kubernetes version upgrade some nodes become NotReady Small clusters (less than 3 nodes) where a single NotReady node impacts core services

Core Knowledge: Kubelet Node‑Status Mechanism

Kubelet reports its node status to the API server every nodeStatusReportFrequency (default 10 s). If reporting fails continuously beyond a threshold the node controller marks the node Unknown or NotReady. The controller logic:

Kubelet reports every 10 s (configurable via --node-status-update-frequency)

If no update for 40 s the node controller sets Unknown If Unknown persists for pod-eviction-timeout (default 5 min) pods are evicted

If the node remains unhealthy for termination-duration (default 5 min) the node is finally marked NotReady The Ready condition is a composite of several sub‑conditions (MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable, kubeletReady). Any sub‑condition being True makes the node NotReady.

Conditions:
  Type               Status
  MemoryPressure    False
  DiskPressure      False
  PIDPressure       False
  NetworkUnavailable False
  kubeletReady      True  <-- calculated by kubelet

Troubleshooting Overview

When a node appears NotReady, follow the ordered checklist below. Adjust the order based on the observed symptoms.

1. Confirm NotReady phenomenon and affected scope
   └─ kubectl get nodes -o wide
   └─ kubectl describe node <code>node-name</code>
2. Check kubelet service status
   └─ systemctl status kubelet
   └─ journalctl -u kubelet -n 200 --no-pager
3. Check container runtime (containerd or docker)
   └─ systemctl status containerd / docker
   └─ crictl info or docker info
4. Inspect node resources (disk, memory, CPU)
   └─ df -h
   └─ free -m
   └─ uptime / top
5. Test network connectivity
   └─ ping / curl API server health endpoint
   └─ check DNS, CNI plugins, routes
6. Check certificate expiration
   └─ kubeadm certs check-expiration
   └─ openssl x509 -in /var/lib/kubelet/pki/cert.crt -noout -dates
7. Identify common root‑cause scenarios and apply fixes
8. Verify node returns to Ready and workloads recover
9. Implement preventive measures and daily health checks

Step 1 – Confirm NotReady Phenomenon and Scope

List all nodes and filter those not Ready:

# List all nodes
kubectl get nodes -o wide

# Show only NotReady nodes
kubectl get nodes | grep -v Ready

# Detailed view of a specific node
kubectl describe node <code>node-name</code>

In the Conditions section of the describe output, look for the failing condition (e.g., DiskPressure=True and kubeletReady=False) and the accompanying message such as "low disk space".

Step 2 – Check Kubelet Process

2.1 View kubelet Service Status

# View kubelet service status (systemd managed)
sudo systemctl status kubelet

If the output shows Active: failed, the kubelet failed to start and logs must be inspected.

2.2 Inspect kubelet Logs

# Recent 300 lines
sudo journalctl -u kubelet -n 300 --no-pager

# Full startup log
sudo journalctl -u kubelet --no-pager | head -500

# Logs after a specific time (replace with actual timestamp)
sudo journalctl -u kubelet --since "2026-04-29 09:00:00" --no-pager

# Search for errors
sudo journalctl -u kubelet -b --no-pager | grep -i error

Typical error categories:

Container‑runtime connection failure – mismatched --container-runtime-endpoint or config file errors.

Certificate problems – missing or expired client certificates.

etcd connection timeout – network or etcd health issues.

Step 3 – Check Container Runtime

3.1 Check containerd Service Status

# Service status
sudo systemctl status containerd

# Recent logs
sudo journalctl -u containerd -n 200 --no-pager

# Verify process existence
ps aux | grep containerd | grep -v grep

3.2 Use crictl to Query Runtime

# Runtime info
sudo crictl info

# List containers (including stopped)
sudo crictl ps -a

# List images
sudo crictl images

If crictl info succeeds but kubelet still reports NotReady, the issue is likely the communication endpoint between kubelet and containerd.

3.3 Check Docker Service (if Docker is used)

sudo systemctl status docker

# Docker logs
sudo journalctl -u docker -n 200 --no-pager

# List containers
sudo docker ps -a

# Docker info (focus on storage driver and cgroup driver)
sudo docker info

Ensure the cgroup driver reported by Docker matches the one configured for kubelet (usually systemd).

Step 4 – Check Node Resources

4.1 Disk Space

# Show usage
df -h

Kubelet’s DiskPressure threshold defaults to less than 10 % free space (configurable via --eviction-hard). When disk usage exceeds 90 % the node becomes NotReady. Clean up space using log rotation, ctr -n k8s.io images prune, or manual removal of large files. Do not delete files under /var/lib/containerd/overlay2 directly; use the containerd prune command.

# Identify large directories
sudo du -sh /var/lib/containerd/* 2>/dev/null | sort -rh | head -10
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -10

# Clean unused images
sudo ctr -n k8s.io images prune -f

# Vacuum old journal logs (keep last 500 MB)
sudo journalctl --vacuum-size=500M

# Verify free space
df -h

# Wait for kubelet to re‑evaluate (1‑2 min)
sleep 120
kubectl get node <code>node-name</code>

4.2 Memory

# Show memory
free -m

# Top memory‑hungry processes
ps aux --sort=-%mem | head -20

# Kubelet memory usage
ps -p $(pgrep kubelet) -o pid,vsz,rss,comm

If available memory is below 10 % of total, the node is under memory pressure. The Linux OOM killer may terminate kubelet, leading to NotReady.

4.3 CPU Load

# Load averages
uptime

# Detailed CPU usage
top -bn1 | head -30

Extreme CPU starvation can prevent kubelet heartbeats, but CPU pressure alone rarely causes NotReady.

4.4 Comprehensive Diagnostic Script

#!/bin/bash
# Basic node health snapshot
uname -r
cat /etc/os-release
nproc
uptime
free -m
df -h | grep -v tmpfs
echo "--- top processes ---"
ps aux --sort=-%cpu | head -20
echo "--- kubelet status ---"
systemctl is-active kubelet
journalctl -u kubelet -n 20 --no-pager | grep -i error

Step 5 – Check Network Connectivity

5.1 Ping API Server from the Node

# Resolve API endpoint from kubeconfig
APISERVER=$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.server}')

echo "API Server: $APISERVER"

# Test HTTPS healthz (skip cert verification if needed)
curl -sk --max-time 5 ${APISERVER}/healthz

5.2 Test DNS Resolution

# Verify cluster DNS
ping -c 3 kubernetes.default.svc

# Verify CoreDNS pods are running
kubectl get pods -n kube-system -l k8s-app=coredns

5.3 Check Node Network Interfaces and Routes

# Interface configuration
ip addr

# Routing table
ip route

# Verify route to API server IP exists
ip route | grep $(echo $APISERVER | awk -F/ '{print $3}' | awk -F: '{print $1}')

5.4 Verify CNI Plugin Status

# Flannel example
ip addr | grep flannel
kubectl logs -n kube-system -l app=flannel --tail=50

# Calico example
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide
kubectl logs -n kube-system -l k8s-app=calico-node --tail=50

Step 6 – Check Certificate Expiration

6.1 Check kubelet Certificate

# kubeadm helper (checks control‑plane certs, not kubelet)
sudo kubeadm certs check-expiration --cert-dir /etc/kubernetes/pki

# Direct check of kubelet cert
sudo openssl x509 -in /var/lib/kubelet/pki/cert.crt -noout -dates

If the kubelet certificate is expired or near expiry, renew it. Renewal will restart kubelet and temporarily evict pods.

6.2 Renew kubelet Certificate

# Option 1: kubeadm renewal (recommended)
sudo kubeadm alpha certs renew kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf

# Option 2: delete old certs and let kubelet request new ones
sudo systemctl stop kubelet
sudo rm -rf /var/lib/kubelet/pki/*
sudo rm -rf /etc/kubernetes/pki/kubelet-client-latest
sudo systemctl start kubelet

Step 7 – Common Root‑Cause Scenarios and Fixes

Scenario 1 – Disk Space Exhaustion (most common)

# Identify large directories
sudo du -sh /var/lib/containerd/* 2>/dev/null | sort -rh | head -10
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -10

# Clean unused images
sudo ctr -n k8s.io images prune -f

# Vacuum old journal logs
sudo journalctl --vacuum-size=500M

# Verify free space
df -h

# Wait for kubelet to re‑evaluate
sleep 120
kubectl get node <code>node-name</code>

Scenario 2 – kubelet OOM Kill

# Find OOM records
sudo dmesg | grep -i "out of memory" | grep -i kubelet

# Reduce memory pressure (evict non‑critical pods, scale down workloads)

# Lower kubelet OOM score so it is less likely to be killed
sudo sed -i 's/^Environment=.*/Environment="KUBELET_OPTS=--oom-score-adj=-999"/' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
sudo systemctl daemon-reload && sudo systemctl restart kubelet

Scenario 3 – Container‑Runtime Misconfiguration

# Verify cgroup driver consistency
grep SystemdCgroup /etc/containerd/config.toml
grep cgroupDriver /var/lib/kubelet/config.yaml

# Align both to "systemd"
# Restart services
sudo systemctl restart containerd
sudo systemctl restart kubelet

Scenario 4 – Kernel or System‑Level Bugs

Out‑of‑date kernel or known bugs (e.g., overlay2 issues, cgroup v2 incompatibilities) can cause node health degradation. Upgrade the kernel or reinstall the OS. If the root filesystem becomes read‑only, a reboot is required after ensuring pod eviction safety.

Scenario 5 – kubelet Configuration Errors

# Validate YAML syntax
yamllint /var/lib/kubelet/config.yaml

# Correct common fields (cgroupDriver, evictionHard, containerLogMaxSize, etc.)
# Restart kubelet after fixing
sudo systemctl restart kubelet

Scenario 6 – etcd Connectivity Problems

If etcd latency or outage prevents kubelet from writing node status, restore etcd health first. Once etcd is healthy, nodes will automatically recover.

Step 8 – Verify Repair Results

8.1 Node Status

# Confirm Ready state
kubectl get nodes -o wide
kubectl describe node <code>node-name</code> | grep -A 20 "Conditions"

8.2 Pod Scheduling

# Ensure evicted pods are rescheduled
kubectl get pods -o wide --all-namespaces | grep <code>node-name</code>

# Look for any remaining Pending or CrashLoopBackOff pods
kubectl get pods --all-namespaces | grep -v Running | grep -v Completed

8.3 Business Functionality

# Verify deployments have expected replica counts
kubectl get deployment -A

# Check services and ingress health
kubectl get svc -A
curl -sk https://<code>ingress-endpoint</code>/health -w "
%{http_code}
"

8.4 Node Resource Validation

# Remote checks
ssh <code>node-name</code> "df -h / && free -m && uptime"

Step 9 – Preventive Measures and Daily Checks

9.1 Daily Node Health Script

#!/bin/bash
NODES=$(kubectl get nodes -o jsonpath='{.items[*].metadata.name}')
ALERT=""
for NODE in $NODES; do
  STATUS=$(kubectl get node $NODE -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')
  if [ "$STATUS" != "True" ]; then
    ALERT+="
[node NotReady] $NODE"
    kubectl describe node $NODE | grep -A 5 "Conditions" >> /var/log/k8s-health.log
  fi
  DISK=$(ssh $NODE "df -h / | tail -1 | awk '{print \$5}' | tr -d %")
  if [ $DISK -gt 85 ]; then
    ALERT+="
[disk pressure] $NODE: $DISK%"
  fi
  MEM_AVAIL=$(ssh $NODE "free -m | awk '/Mem:/ {print \$7}'")
  MEM_TOTAL=$(ssh $NODE "free -m | awk '/Mem:/ {print \$2}'")
  MEM_USAGE=$(( (MEM_TOTAL-MEM_AVAIL)*100/MEM_TOTAL ))
  if [ $MEM_USAGE -gt 90 ]; then
    ALERT+="
[memory pressure] $NODE: $MEM_USAGE%"
  fi
  KUBELET=$(ssh $NODE "systemctl is-active kubelet")
  if [ "$KUBELET" != "active" ]; then
    ALERT+="
[kubelet down] $NODE: $KUBELET"
  fi
done
if [ -n "$ALERT" ]; then
  echo -e "K8s Node Health Alert:$ALERT" | tee -a /var/log/k8s-health.log
else
  echo "$(date): All nodes healthy" >> /var/log/k8s-health.log
fi

9.2 Tune kubelet Eviction Thresholds

# /var/lib/kubelet/config.yaml example
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "5%"
  imagefs.available: "10%"
evictionSoft:
  memory.available: "200Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "2m"
  nodefs.available: "2m"
  imagefs.available: "2m"
evictionPressureTransitionPeriod: "2m"

9.3 Prometheus Alerts for Node Pressure

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: node-disk-alerts
  namespace: monitoring
spec:
  groups:
  - name: node-resources
    rules:
    - alert: NodeDiskPressure
      expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.15
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Node {{ $labels.instance }} has less than 15% disk space"
        description: "Disk usage is {{ $value | humanizePercentage }}"
    - alert: NodeDiskPressureCritical
      expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.05
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Node {{ $labels.instance }} has less than 5% disk space"
        description: "Disk usage is {{ $value | humanizePercentage }}"

9.4 Automatic Certificate Renewal

# Monthly cron on masters
0 3 1 * * /usr/bin/kubeadm certs renew all --kubeconfig=/etc/kubernetes/admin.conf

9.5 Limit Pods per Node

# Verify max‑pods setting (default 110)
ps aux | grep kubelet | grep max-pods
# Enforce in kubelet config if needed
maxPods: 110

Conclusion

Diagnosing a Kubernetes NotReady node requires understanding the kubelet status‑reporting loop and systematically checking kubelet health, container runtime, node resources, network connectivity, and certificates. The most frequent root causes are disk‑space exhaustion, kubelet OOM kills, and runtime misconfiguration. After fixing the issue, always verify node readiness, pod scheduling, and business functionality before declaring success. Implement daily health checks, proper eviction thresholds, monitoring alerts, and certificate automation to prevent recurrence.