Troubleshoot Pod Frequent Restarts in 60 Seconds: OOMKilled vs Misconfigured Health Checks

A new version deployment caused a data-collector pod to restart 28 times in two hours (roughly every four minutes). The author demonstrates a systematic three-step troubleshooting process.

Step 1: Check Pod State to Identify the Cause

Run kubectl describe pod and examine State and Last State. In the example, Last State shows Reason: OOMKilled and Exit Code: 137. Two key exit codes to remember:

137 : Process received SIGKILL, typically an OOM kill.

143 : Process received SIGTERM, usually a graceful shutdown or liveness probe failure.

Step 2: Examine Events for Health Check Issues

Scroll to the Events section. If you see Liveness probe failed with messages like context deadline exceeded followed by Container failed liveness probe, will be restarted, the restart is due to a misconfigured liveness probe.

Step 3: Inspect Previous Container Logs

Use kubectl logs data-collector-xxx --previous to view logs from the crashed container instance. If this returns nothing, the pod was likely OOM-killed instantly before logs could be flushed to disk.

Resolving OOMKilled (Memory Limit Exceeded)

Immediate mitigation: increase the memory limit in the pod spec.

resources:
  limits:
    memory: 2Gi
  requests:
    memory: 512Mi

Root causes to investigate:

Memory leaks : unclosed database connections, file streams, or large objects retained indefinitely.

Java heap misconfiguration : -Xmx set higher than the container memory limit.

Traffic spikes : sudden large data loads into memory.

Resolving Liveness Probe Failures

The most common mistake is setting initialDelaySeconds too short. If the application needs 30 seconds to start but the probe begins at 10 seconds, it will fail repeatedly, causing a restart loop. Example corrected configuration:

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

Key probe parameters and suggested values: initialDelaySeconds — Wait time after startup before first probe. Suggested: 30-60s (match startup time). periodSeconds — Interval between probes. Suggested: 10-30s. timeoutSeconds — Probe timeout threshold. Suggested: 3-5s. failureThreshold — Consecutive failures before restart. Suggested: 3-5.

Critical Reminders

Do not confuse liveness and readiness probes. Liveness failure triggers a restart; readiness failure only removes the pod from service endpoints. Swapping them causes either constant restarts or traffic sent to unhealthy pods.

Liveness probes must not depend on external resources. Using a database connection check as a liveness probe is dangerous: if the database is down, restarting the pod will not help and may create a restart storm.

Handling missing logs after OOM. Java applications often cannot flush logs before SIGKILL. Add JVM option -XX:+ExitOnOutOfMemoryError or use an asynchronous logging framework to ensure logs are written before termination.

Quick Three-Step Diagnosis Cheat Sheet

kubectl describe pod

→ check Reason (OOMKilled vs Liveness probe failed). kubectl logs --previous → examine crash logs; if empty, suspect instant OOM. kubectl top pod → observe memory usage relative to limit; if near limit, increase limit or optimize code; if normal, review probe configuration.

Symptom → Root Cause → Action Reference

Symptom: Reason: OOMKilled | Root Cause: Memory limit exceeded | Action: Increase limit / investigate memory leak / optimize code

Symptom: Exit Code: 137 | Root Cause: Process killed by SIGKILL | Action: Same as OOM, investigate memory

Symptom: Liveness probe failed | Root Cause: Health check failing | Action: Increase initialDelaySeconds / verify probe path

Symptom: Slow startup, repeated restarts | Root Cause: initialDelaySeconds too short | Action: Allow sufficient startup wait time

Symptom: logs --previous shows nothing | Root Cause: Instant OOM, no log flush | Action: Configure async logging + OOM JVM flags

Symptom: Liveness and readiness confused | Root Cause: Misconfigured probes | Action: Liveness controls restart, readiness controls traffic