How to Diagnose and Fix Constant Kubernetes Pod Restarts (OOM)

Summary : Kubernetes issue diagnosis – continuous pod restarts, how to locate the cause and resolve it.

When performing performance testing, we often find our pod services repeatedly restart; using kubectl get pods we can gradually locate the problem.

Phenomenon : a running pod restarts too many times in a short period.

Problem identification method : view pod logs.

kubectl get event                           # view events from the past hour
kubectl describe pod pod_name               # describe the current pod
kubectl logs -f pod_name --previous          # view the pod logs before the restart, starting from that point

###############
Usually the above three commands are sufficient.
In this case we used the following command to solve the issue:

kubectl describe pod pod_name

Problem cause : OOM, pod was killed and restarted (insufficient memory).

View the service's deployment.yaml file

We discovered that the deployment.yaml set memory limits for the service.

Solution : increase the memory limit value, then delete the old resources with kubectl delete -f yaml and apply the updated configuration with kubectl apply -f yaml.

Thus we successfully solved the problem and identified the root cause.

Link: https://bbs.huaweicloud.com/blogs/413818

© Huawei Cloud Community. All rights reserved. Original author.