Kubernetes Troubleshooting in Practice: 20 Survival Rules from Real Incidents

Incident Overview

At 02:17 a massive alert on an e‑commerce platform “FastShop Mall” showed order‑service success rate drop from 99.97% to 72%, gateway 502 errors, pods restarting, and the database connection pool exhausted. The system handled 23 million daily active users, a peak of 85 k QPS, more than 200 microservices, three Kubernetes clusters with 260 nodes, and about 1.4 billion RocketMQ messages per day. Stack: Nginx + Spring Cloud Gateway, Spring Boot services registered in Nacos, Redis Cluster, RocketMQ, MySQL 8 sharded, observability via Prometheus, Grafana and EFK, and Kubernetes 1.27 with containerd.

Post‑mortem identified the root cause as a seemingly harmless ConfigMap update combined with an overly short terminationGracePeriodSeconds, aggressive readiness/liveness probes and an HPA scaling‑down policy, which together triggered a rolling‑update avalanche.

Unified Diagnostic Model

Kubernetes is declarative; most failures can be expressed as “desired state recorded but actual state not converged”. Diagnose by walking the control chain:

Did the resource object reach the API server?

Did the controller generate downstream resources?

Did the scheduler place the pod?

Did kubelet on the target node succeed?

Are container, network and storage usable?

Typical “5‑minute triage” commands:

kubectl get pods -A
kubectl get events -A --sort-by=.lastTimestamp | tail -n 50
kubectl get nodes
kubectl top pods -A

20 Survival Rules (grouped)

Node & OS (Rules 1‑4)

Rule 1 – Node NotReady : Check kubectl get nodes, kubectl describe node, kubelet status, and look for NodeDiskPressure, NodeMemoryPressure, certificate expiry, CSI failures or OOM.

Rule 2 – Disk pressure from log rotation : A full /var triggers imagefs / nodefs eviction; clean the disk and restart containerd.

Rule 3 – CSI / storage topology : Verify PVC binding, VolumeAttachment status and that the node’s zone matches the volume’s zone; use volumeBindingMode: WaitForFirstConsumer for multi‑zone clusters.

Rule 4 – Prevent node‑wide disk exhaustion : Limit emptyDir size, rotate logs, and monitor node‑disk usage per pod.

Pod Scheduling (Rules 5‑9)

Rule 5 – Pending ≠ lack of resources : kubectl describe pod may show messages like “0/26 nodes are available: 5 Insufficient cpu, 10 Insufficient memory…”. Adjust requests based on historical usage rather than total capacity.

Rule 6 – etcd latency : Slow etcdctl endpoint health or log lines “etcd request took too long” indicate a control‑plane bottleneck; keep etcd on dedicated SSDs and monitor its size.

Rule 7 – Ingress 502/503 : Verify DNS resolution, Service/Endpoint mapping, and that port names match target ports.

Rule 8 – Cross‑namespace service discovery : Ensure the namespace and service exist, the pod can reach kube‑dns, and CoreDNS has sufficient CPU/memory.

Rule 9 – Service‑mesh sidecar readiness : Istio sidecars may not be ready, causing 503s even though the application container runs; check istio-proxy health and istioctl proxy-status.

Scheduling & Lifecycle (Rules 10‑13)

Rule 10 – CrashLoopBackOff : Often caused by missing dependencies (Redis, MySQL, config). Use initContainers to wait for dependencies or a proper startupProbe.

Rule 11 – Graceful termination : preStop must stop traffic before draining; ensure terminationGracePeriodSeconds is long enough for in‑flight requests.

Rule 12 – HPA inactivity : kubectl get hpa showing unknown/100m usually means metrics‑server is down or pods lack CPU requests.

Rule 13 – Affinity & tolerations : Rigid requiredDuringSchedulingIgnoredDuringExecution can lock services to a small node pool; prefer preferredDuringScheduling and keep a fallback pool.

Storage & State (Rules 14‑16)

Rule 14 – ContainerCreating stalls : Check CSI driver health, volume binding mode, and node‑zone compatibility when a pod stays in ContainerCreating.

Rule 15 – StatefulSet stability : Deleting a pod does not delete its data. Recover from snapshots or adjust preStop and termination grace before deletion.

Rule 16 – emptyDir usage : Treat emptyDir as node‑local cache; set sizeLimit and rotate logs to avoid node‑disk exhaustion.

Application & Configuration (Rules 17‑19)

Rule 17 – ConfigMap/Secret updates : Updating the object does not guarantee the application reloads. Use directory mounts (avoid subPath) and implement hot‑reload or rolling updates.

Rule 18 – Java OOMKilled : Container OOM is often caused by total memory (heap + Metaspace + direct memory). Use -XX:MaxRAMPercentage and reserve headroom for native allocations.

Rule 19 – livenessProbe misuse : Probes should only detect process death. External‑dependency checks belong in readinessProbe or an application health endpoint.

Observability (Rule 20)

Dashboards that only show CPU and memory are useless during incidents. A practical panel must correlate:

Business signals – latency, error rate, traffic saturation.

Kubernetes signals – pod restarts, evictions, probe failures.

Link‑level signals – logs, traces, downstream latency.

Linking these three layers in Grafana reduced mean‑time‑to‑locate a fault from 45 minutes to 8 minutes.

Key Incident Timeline

02:15 Ops updated order‑service ConfigMap – no immediate impact.

02:16 Deployment started rolling update – old pods entered termination.

02:17 Gateway returned 502 – success rate dropped.

02:17:30 New pods not ready, HPA scaled down – remaining healthy pods overloaded.

02:18 Order‑service CPU spiked, upstream timeouts – massive order failures.

Checklist for Quick Diagnosis

After a release or traffic shift

Did a Deployment, ConfigMap, Secret or Ingress change just now?

Is readinessProbe passing too early?

Did preStop and graceful termination actually run?

Are client connection pools still using old endpoints?

Is HPA scaling up and down simultaneously?

When a pod cannot start

Identify the pod phase: Pending, ContainerCreating, CrashLoopBackOff or OOMKilled.

Check Events for scheduling, mounting or probe failures.

Verify node health – DiskPressure, MemoryPressure.

Confirm request/limit values match real load.

When a service call fails

Validate DNS, Service, Endpoint and Ingress port mappings.

Check NetworkPolicy for blocked DNS or DB traffic.

Inspect Service Mesh timeout/retry settings.

Monitor conntrack usage on the node.

When a stateful service misbehaves

Determine whether the issue lies in the Pod, PVC, CSI driver or underlying storage topology.

Ensure StatefulSet data has backup and recovery procedures.

Avoid treating “delete pod” as a data‑recovery action.

Takeaways

Identify which control‑plane layer the problem originates from.

Distinguish root cause from downstream ripple effects.

Prioritize fixing traffic routing and lifecycle handling before resorting to pod restarts.

Automation only works on top of reliable monitoring and sane resource declarations.

A mature platform prevents repeatable failures, not just recovers from them.