Mastering K8s Troubleshooting: Common Production Issues and Essential Commands

Problem Background and Scope

Kubernetes cluster stability directly impacts service availability. Common production‑level symptoms include pod startup failures, node unavailability, service access errors, storage mount failures, and permission issues.

Troubleshooting Principles

Layered investigation : start from the node, then the pod, then the application.

Check status before logs : kubectl get pod and kubectl describe pod often reveal the root cause.

Avoid blind restarts : restarting a pod can lose context; investigate first.

Backup configurations before changes : e.g.

kubectl get deployment <name> -n <ns> -o yaml > backup.yaml

.

Common Pod Issues

CrashLoopBackOff

Symptoms : pod repeatedly restarts with increasing back‑off intervals (10s, 20s, 40s …).

kubectl get pod -n <namespace>
NAME                     READY   STATUS            RESTARTS   AGE
my-app-5d8f9c6b4-x7r2k   0/1     CrashLoopBackOff   3          2m15s

Typical causes : configuration errors, wrong start command, missing dependencies, failing livenessProbe, insufficient permissions, missing libraries, OOM kill.

Inspect events and status

kubectl describe pod <pod-name> -n <namespace>

Look for Last State: Terminated and Reason: OOMKilled.

View previous container logs

# Last terminated container logs
kubectl logs <pod-name> -n <namespace> --previous
# For multi‑container pods
kubectl logs <pod-name> -n <namespace> -c <container> --previous

Verify missing config files, connection errors, etc.

Check resource limits

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'

Ensure requests.memory and limits.memory are appropriate; for memory‑intensive workloads set limits.memory to 1.5‑2× requests.memory.

Validate livenessProbe

# Example failing event
Warning  Unhealthy  2m ago  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 503
# Probe settings
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].livenessProbe}'
# Recommended values
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

Enter the container for deeper inspection

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
# Common checks inside the container
ps aux
ls -la /proc/1/fd
cat /proc/net/tcp
cat /etc/resolv.conf
ping -c 3 kubernetes.default.svc
curl -v http://localhost:<port>/health

Verify fix

watch kubectl get pod <pod-name> -n <namespace>

Ensure READY is 1/1 and STATUS is Running.

ImagePullBackOff / ErrImagePull

Symptoms : pod cannot pull the container image.

kubectl get pod -n <namespace>
NAME                     READY   STATUS            RESTARTS   AGE
my-app-5d8f9c6b4-x7r2k   0/1     ImagePullBackOff   0          5m

Inspect pod events

kubectl describe pod <pod-name> -n <namespace>

Look for Failed to pull image or ErrImagePull.

Verify image name and tag

# Try pulling directly on a node
docker pull <image-name>
# Or with containerd
crictl pull <image-name>

Errors such as manifest unknown indicate a wrong tag.

Check imagePullSecrets

# List existing Docker secrets
kubectl get secrets -n <namespace> | grep -i docker
# Create or update secret
kubectl create secret docker-registry <secret-name> \
  --docker-server=<registry-server> \
  --docker-username=<username> \
  --docker-password=<password> \
  --docker-email=<email> -n <namespace>
# Patch deployment to use the secret
kubectl patch deployment <name> -n <namespace> -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"<secret-name>"}]}}}'

Inspect node‑level network

# On the node
journalctl -u kubelet --no-pager | grep -E "ImagePullBackOff"
nslookup registry.example.com
nc -zv registry.example.com 443
curl -v https://registry.example.com/v2/

Pending

Symptoms : pod stays in Pending without errors.

kubectl get pod -n <namespace>
NAME                     READY   STATUS    RESTARTS   AGE
my-app-5d8f9c6b4-x7r2k   0/1     Pending   0          10m

Check pod events

kubectl describe pod <pod-name> -n <namespace>

Look for Unschedulable with reasons such as Insufficient memory or node(s) had taints that the pod didn't tolerate.

Inspect cluster resources

# Requires metrics‑server
kubectl top nodes
# If metrics‑server not installed
kubectl describe nodes | grep -A5 "Allocated resources"
kubectl get pods -n <namespace> -o wide

Check taints and tolerations

# Node taints
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.taints[*].key}{"
"}{end}'
# Pod tolerations
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.tolerations}'
# Affinity rules
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.affinity}'

Verify PVC status (if pod uses storage)

kubectl get pvc -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

Terminating / ContainerCreating Stuck

Symptoms : pod remains in Terminating or ContainerCreating for minutes.

kubectl get pod -n <namespace>
NAME                     READY   STATUS        RESTARTS   AGE
my-app-5d8f9c6b4-x7r2k   1/1     Terminating   0          30m

Inspect pod YAML for finalizers

kubectl get pod <pod-name> -n <namespace> -o yaml

Look for metadata.finalizers and deletionTimestamp.

Check kubelet logs on the node

journalctl -u kubelet --no-pager | grep -E "(pod|%s)" | tail -100

Inspect container processes

# On the node
crictl ps -a | grep <pod-name>
ps aux | grep -E "(zombie|defunct)"

Check volume mounts

# List volumes used by the pod
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.volumes[*].name}'
# On the node
mount | grep <pvc-name>
df -h | grep <pvc-name>

Force delete if necessary

# Remove finalizers
kubectl patch pod <pod-name> -n <namespace> -p '{"metadata":{"finalizers":null}}' --type=merge
# Force delete with short grace period
kubectl delete pod <pod-name> -n <namespace> --grace-period=5 --force

Service and Network Issues

Endpoints Missing

Symptoms : Service exists but Endpoints are empty, causing connection failures.

# Service
kubectl get svc my-service -n <namespace>
# Endpoints
kubectl get endpoints my-service -n <namespace>
NAME        ENDPOINTS   AGE
my-service   <none>      30d

Check Service selector

kubectl get svc my-service -n <namespace> -o jsonpath='{.spec.selector}'

Verify pod labels

kubectl get pods -n <namespace> --show-labels | grep <label-key>

Test label selection

kubectl get pods -n <namespace> -l "<key>=<value>"

If no pods are returned, adjust the selector or pod labels.

Fix

# Patch Service selector
kubectl patch svc my-service -n <namespace> -p '{"spec":{"selector":{"app":"<correct-label>"}}'
# Or patch pod label (temporary)
kubectl label pod <pod-name> -n <namespace> --overwrite app=<correct-label>

DNS Resolution Failures

Symptoms : pods cannot resolve service names (e.g., could not resolve host).

# Inside a pod
nslookup my-service
curl http://my-service

Check CoreDNS / kube‑dns pod status

kubectl get pods -n kube-system -l k8s-app=kube-dns
# or
kubectl get pods -n kube-system -l app.kubernetes.io/name=coredns

Inspect DNS pod logs

kubectl logs -n kube-system <coredns-pod> --tail=50

Inspect pod /etc/resolv.conf

kubectl exec -it <pod> -n <ns> -- cat /etc/resolv.conf

Expected entry: nameserver 10.96.0.10 (cluster DNS IP).

Test DNS query from DNS pod itself

kubectl exec -n kube-system <coredns-pod> -- nslookup kubernetes.default

Check NetworkPolicy rules that might block port 53

kubectl get networkpolicy -n <namespace>
kubectl describe networkpolicy <policy> -n <namespace>

Fix

# Restart DNS deployment
kubectl rollout restart deployment/coredns -n kube-system

NetworkPolicy Misconfiguration

Symptoms : pod cannot reach a Service or other pods after a NetworkPolicy is applied.

Review the NetworkPolicy ingress and egress rules .

Temporarily delete the policy to confirm impact

kubectl delete networkpolicy <policy-name> -n <namespace>

Re‑create a correct whitelist policy (example below)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: <namespace>
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}

Ingress Failures

Symptoms : external requests to an Ingress hostname return 404 or timeout.

Verify Ingress controller pods are running

kubectl get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx

Inspect the Ingress resource

kubectl get ingress -n <ns>
kubectl describe ingress <name> -n <ns>

Confirm IngressClass matches the deployed controller .

Check backend Service and Endpoints

kubectl get svc -n <ns>
kubectl get endpoints <svc> -n <ns>

From the controller pod, curl the backend Service

kubectl exec -n ingress-nginx <controller-pod> -- curl -v http://<backend-svc>.

Storage Issues

PVC Pending

Symptoms : PersistentVolumeClaim stays in Pending, preventing pod startup.

kubectl get pvc -n <ns>
NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS
data-myapp  Pending               20Gi       RWO            slow-storage

View PVC events for the exact error

kubectl describe pvc <pvc-name> -n <ns>

Ensure the referenced StorageClass exists

kubectl get storageclass
# Create missing class if needed
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Check CSI driver pods and logs

kubectl get pods -n kube-system | grep -E "csi|storage"
kubectl logs -n kube-system <csi-pod> --tail=100

Verify accessModes match available PVs .

If using WaitForFirstConsumer, ensure a pod is scheduled to trigger binding .

Volume Mount Failures

Symptoms : pod stuck in ContainerCreating with mount errors.

Events:
  Warning  FailedMount  2m ago  kubelet  MountVolume.SetUp failed for volume "pvc-xxx": rpc error: code = Internal desc = could not mount disk: Format disk: exit status 1

On the node, check existing mounts and kubelet logs

mount | grep pvc
journalctl -u kubelet --no-pager | grep -E "MountVolume|FailedMount" | tail -50

Inspect filesystem state (lsblk, df, fdisk) on the node .

Check CSI driver logs for the node plugin .

Remediation

Re‑format or replace the underlying storage (data loss risk).

Fix storage backend health issues.

Delete and recreate the pod after storage is healthy.

Authentication, Authorization, and Scheduling

RBAC Permission Issues

Pods or ServiceAccounts receive forbidden errors.

# Test permission
kubectl auth can-i get pods --as=system:serviceaccount:<ns>:<sa>

Identify the ServiceAccount used by the pod ( spec.serviceAccountName).

Check existing Role/ClusterRole and RoleBinding/ClusterRoleBinding that bind it.

If missing, create a Role and bind it:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-role
  namespace: <ns>
spec:
  rules:
  - apiGroups: [""]
    resources: ["pods","services"]
    verbs: ["get","list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-rolebinding
  namespace: <ns>
subjects:
- kind: ServiceAccount
  name: <sa>
  namespace: <ns>
roleRef:
  kind: Role
  name: my-role
  apiGroup: rbac.authorization.k8s.io

ServiceAccount Problems

Pod cannot access the API server despite using a ServiceAccount.

# Verify token existence
kubectl exec -it <pod> -n <ns> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
# Test API call from inside pod
APISERVER=https://kubernetes.default.svc
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -sk $APISERVER/api --header "Authorization: Bearer $TOKEN"

Taints and Tolerations

Pod cannot be scheduled due to node taints.

# View pod events for Unschedulable
kubectl describe pod <pod> -n <ns> | grep -i taint
# List node taints
kubectl get node <node> -o jsonpath='{.spec.taints}'
# Add toleration to pod spec
kubectl patch deployment <name> -n <ns> -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"<taint-key>","operator":"Exists","effect":"NoSchedule"}]}}}}'

Production Troubleshooting Workflow

Quick Diagnosis Path

# 1. List pods and note STATUS
kubectl get pods -n <ns>
# 2. Describe the problematic pod
kubectl describe pod <pod> -n <ns>
# 3. View previous container logs
kubectl logs <pod> -n <ns> --previous
# 4. Check resource usage
kubectl top pod -n <ns>
# 5. Follow the specific troubleshooting guide for the observed status (CrashLoopBackOff, ImagePullBackOff, Pending, etc.)

Layered Investigation

Cluster layer : nodes, namespaces, cluster‑wide events.

Network layer : Services, Endpoints, NetworkPolicy, DNS, pod‑to‑pod connectivity.

Storage layer : PVC/PV, StorageClass, CSI drivers.

Application layer : pod spec, container logs, in‑container debugging.

Useful kubectl Debug Tricks

# Show diff between live config and a manifest
kubectl diff -f resource.yaml
# Watch resources in real time
watch kubectl get pods -n <ns>
# Find who created a pod
kubectl get pod <pod> -n <ns> -o jsonpath='{.metadata.ownerReferences}'
# List all labels of a pod
kubectl get pod <pod> -n <ns> --show-labels
# Filter pods by label
kubectl get pods -n <ns> -l "app=myapp,tier=frontend"
# Rollout history and rollback
kubectl rollout history deployment <name> -n <ns>
kubectl rollout undo deployment <name> -n <ns>

Best Practices

Logging and Monitoring

Centralized logging : ship all pod logs to a log platform (e.g., ELK, Loki) instead of relying on kubectl logs for production debugging.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Log_Level    info
        Daemon       off
    [INPUT]
        Name         tail
        Path         /var/log/containers/*.log
        Parser       docker
        Tag          kube.*
    [OUTPUT]
        Name         es
        Match        kube.*
        Host         elasticsearch.logging.svc
        Port         9200

Monitoring & alerts : Deploy Prometheus + Grafana. Key alerts include Node pressure (MemoryPressure, DiskPressure, NotReady), Pod CrashLoopBackOff/OOMKilled, PVC pending >5 min, high restart counts, CPU/Memory >80%.

Resource Quota and LimitRange Design

Set ResourceQuota and LimitRange per namespace to prevent resource exhaustion.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: <namespace>
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limit
  namespace: <namespace>
spec:
  limits:
  - type: Container
    default:
      memory: 512Mi
      cpu: 200m
    defaultRequest:
      memory: 256Mi
      cpu: 100m
    max:
      memory: 2Gi
      cpu: "1"

Backup and Rollback

All manifests should live in a Git repository (GitOps). Use kubectl rollout undo for Deployments and define proper RollingUpdate strategies for StatefulSets.

# Deployment rollback
kubectl rollout history deployment <name> -n <ns>
kubectl rollout undo deployment <name> -n <ns>
# StatefulSet rolling update example
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

Production Change Safety

Before any change:

Identify impact scope (pods, services, users).

Backup the resource definition:

kubectl get <resource> <name> -n <ns> -o yaml > backup.yaml

Prepare a rollback plan.

Execute during low‑traffic windows.

Notify stakeholders and follow approval processes.

High‑risk actions (deleting namespaces, modifying RBAC, removing StorageClasses, editing etcd, bulk pod deletions) must be approved and performed with extra caution.

Summary

Kubernetes production troubleshooting relies on systematic, layered diagnosis. From node health to pod status, from network/DNS to storage binding, each layer has specific checks. The most efficient workflow is to examine kubectl get and kubectl describe output first, then dive into logs, resource usage, and component‑specific diagnostics.

Quick Reference List

CrashLoopBackOff : kubectl logs --previous. Common causes – config error, missing dependency, OOM.

ImagePullBackOff : kubectl describe pod (Events). Common causes – image name typo, auth failure, network issue.

Pending : kubectl describe pod (Events). Common causes – insufficient resources, taint mismatch, PVC pending.

Terminating stuck : kubectl get pod -o yaml (finalizers). Common causes – finalizer references, volume cleanup, kubelet‑API issues.

Node NotReady : kubectl describe node (Conditions). Common causes – disk full, memory pressure, network failure, kubelet crash.

Endpoints empty : kubectl get svc -o jsonpath (selector). Common causes – label mismatch, pod not running, wrong namespace.

DNS resolution failure : kubectl logs -n kube-system coredns. Common causes – CoreDNS down, NetworkPolicy block.

PVC Pending : kubectl describe pvc (Events). Common causes – missing StorageClass, CSI error, quota limit.

Key takeaways:

Always inspect status and events before logs.

Backup configurations and keep rollback paths.

Avoid blind restarts; investigate root cause first.

Set both requests and limits for containers.

Robust logging and monitoring are essential for rapid issue detection.

Mastering these troubleshooting patterns enables operators to keep Kubernetes clusters reliable and services available in production environments.