Free 172‑Page Alibaba Cloud Kubernetes Handbook: Deep Dive into 6 Core Principles

Problem Description

A node in an Alibaba Cloud Kubernetes cluster intermittently transitions to NotReady. Restarting the node’s kubelet makes the node Ready for roughly three minutes, after which it reverts to NotReady. The issue is observed roughly once or twice per month per customer and can reappear after about 20 days.

Node‑Readiness Architecture

Node readiness is determined by four components:

etcd (cluster data store)

API Server (cluster entry point)

Node controller (updates node conditions) kubelet (runs on each node) kubelet plays two roles:

Cluster controller: periodically fetches pod specifications from the API Server and creates or deletes pods accordingly.

Node health monitor: gathers node status and reports it to the API Server via the NodeStatus mechanism.

PLEG and NodeStatus Interaction

The NodeStatus condition relies heavily on the Pod Lifecycle Events Generator ( PLEG ). PLEG periodically inspects container states, converts changes into events, and feeds them to the kubelet sync loop. If a PLEG check does not finish within its timeout, NodeStatus treats the node as unhealthy and marks it NotReady.

Default PLEG timing parameters are:

Execution interval: 1 second (time between successive checks)

Timeout: 3 minutes (maximum duration for a single check)

Observed Three‑Minute Ready Window

When kubelet restarts, the first PLEG check often fails to complete. The node remains Ready until the three‑minute timeout expires, at which point the NodeStatus controller synchronizes a NotReady condition to the API Server. This explains the consistent three‑minute readiness period after a kubelet restart.

Root Cause Investigation

The troubleshooting path traversed the Kubernetes stack, the container runtime, sdbus, and finally systemd. A bug in systemd caused the PLEG check to hang, triggering the timeout. The bug has been fixed upstream in systemd, reducing the recurrence of this issue.