Why Are My Kubernetes Pods Getting Evicted? Diagnosing DiskPressure and Log Bloat

Background

Many pods were found in the Evicted state, but without monitoring permissions the author could not check Grafana for anomalies.

The eviction likely occurred when the node’s disk usage exceeded the kubelet’s eviction thresholds; later the space was reclaimed, making the problem appear resolved.

On each Kubernetes node the kubelet’s default root directory is /var/lib/kubelet and the log directory is /var/log. Both reside on the system partition, which is also used by EmptyDir volumes, container logs, image layers and writable layers. Ephemeral‑storage manages this partition.

Large Number of Evicted Pods

kubectl get po -A -o wide | grep -v "Running"
NAMESPACE   NAME   READY   STATUS   RESTARTS   AGE   IP   NODE   ...
nsop   account-service-pre-master-6db67f5cc-5nrgf   0/1   Evicted   0   103m   <none>   node2.pre.ayunw.cn   <none>   <none>
... (additional rows omitted) ...

Even after eviction each pod remains attached to the network and consumes an IP address, which can exhaust a fixed IP pool. Monitoring evicted pods with kubectl get pod becomes difficult without filtering.

Inspecting an Evicted Pod’s Node

Describing a pod shows a warning DiskPressure, indicating disk stress on the node.

kubectl describe po account-service-pre-master-6db67f5cc-5nrgf -n nsop
...
Warning  Evicted  100m  kubelet, node2.pre.ayunw.cn  The node had condition: [DiskPressure].

Checking Node Disk Usage

df -Th | egrep -v "overlay2|kubernetes|docker"
Filesystem   Type   Size  Used Avail Use% Mounted on
/dev/vda1   ext4    50G  7.9G   39G  17% /
/dev/vdb1   xfs   200G 138G   63G  69% /data

Disk usage still shows ample free space, suggesting that space was reclaimed after the eviction event.

Inspecting Disk I/O

iostat -xk 1 3
... (output omitted) ...

No obvious I/O pressure was observed.

Node Logs

tail -500 kubelet.log | grep "Evicted"
tail -500 /var/log/messages | grep "Evicted"

No eviction‑related entries were found in the kubelet or system logs.

Temporary Fixes for Log Bloat

On the master node, locate the node where evicted pods are scheduled, then under /data/kubernetes/kubelet/pods use du -sh to find large pod directories and delete old rotated log files (e.g., skywalking-api.log.2021_08_12_02_56_35 ). Delete the pod; the EmptyDir volume will be removed when the pod restarts.

Permanent Log Retention Solution

Modify the SkyWalking agent configuration to limit retained log files, for example by setting logging.max_history_files=3, and rebuild the agent image.

# Dockerfile snippet
FROM registry.ayunw.cn/library/alpine:3.12.0
ENV LANG=C.UTF-8 SKYWLKING_AGENT_VERSION=8.6.0
...
COPY agent.config /opt/skywalking/agent/config/

# agent.config snippet
logging.max_history_files=${SW_LOGGING_MAX_HISTORY_FILES:3}

Rebuild the image and reference it in the Deployment.

# deployment.yaml excerpt
initContainers:
- name: init-skywalking-agent
  image: "registry.ayunw.cn/library/skywalking-agent:33-ac402d20"
  command: ['sh','-c','set -ex;mkdir -p /skywalking/agent;cp -r /opt/skywalking/agent/* /skywalking/agent;']
  volumeMounts:
  - name: vol-apm-empty
    mountPath: /skywalking/agent
containers:
- name: demo-hello-pre-master
  image: "registry.ayunw.cn/paas/demo-hello:537-c87b6177"
  volumeMounts:
  - name: vol-apm-empty
    mountPath: /skywalking/agent
volumes:
- name: vol-apm-empty
  emptyDir: {}

Although SkyWalking appears to be the main cause of the disk pressure, without proper monitoring the root cause cannot be confirmed definitively; additional observability is recommended.