Why Your Pods Keep CrashLoopBackOff: 8 Root Causes & SRE-Grade Troubleshooting Playbook

1. Core Mechanism: What CrashLoopBackOff Actually Is

CrashLoopBackOff is not an application error but a Kubelet protective exponential backoff state machine. When a main container (PID 1) exits with a non-zero code and restartPolicy is Always (default) or OnFailure, Kubelet's PodSyncLoop restarts it. If crashes recur rapidly, Kubelet pauses restarts with delays: 10s → 20s → 40s → 80s → 160s → 300s (5 min cap) . During this backoff the Pod shows CrashLoopBackOff.

Under Kubernetes v1.37.2 + Containerd 1.7.23 + runc 1.2 (Cgroup v2, SystemdCgroup), the call chain from scheduling to CrashLoopBackOff is:

[API Server 调度 Pod] ---> [Node: Kubelet PodSyncLoop]
         | 
         v (1) CRI gRPC 调用
  [Containerd 1.7.23]
         | 
         v (2) 创建 Cgroup v2 与 Namespace
  [runc 启动容器 PID 1]
         | 
         +-----------------------+-----------------------+
         v                       v                       v
[启动阶段故障]           [运行时内核击杀]           [健康检查失败]
 Exit 1 / 126 / 127       Exit 137 (OOMKilled)      Exit 143 (SIGTERM)
 (配置/权限/架构错误)      (堆外内存/存储超限)       (Liveness 误杀)
         |                       |                       |
         +-----------------------+-----------------------+
                                 |
                                 v (3) 非零退出码上报 Kubelet
                    [指数退避队列: 10s..300s]
                     状态置为 CrashLoopBackOff

1.1 Production Baseline

All YAML, kernel params, PromQL, and commands validated against:

Kubernetes v1.37.2 (pure CRI, DockerShim removed)

Containerd 1.7.23 + runc 1.2, SystemdCgroup (Cgroup v2)

Ubuntu 24.04.1 LTS / Rocky 9.4, Kernel 6.8.0-45 / 5.14.0-427

Prometheus 3.13 LTS + KSM, kube-state-metrics v2.14.0

1.2 Eight Root Causes Quick Reference

1. Memory limit exceeded (OOMKilled) : Exit code 137 / OOMKilled. Trigger: hit Cgroup memory.max. Key diagnostic: dmesg -T.

2. Config/startup error : Exit code 1 or 2 / Error. Trigger: Env/ConfigMap/code exception. Key diagnostic: kubectl logs --previous.

3. Probe kills slow start : Exit code 137 or 143 / Error. Trigger: LivenessProbe timeout sends SIGTERM. Key diagnostic: kubectl describe pod.

4. Image permission/arch mismatch : Exit code 126 / 127 / 255. Trigger: exec format error or missing +x. Key diagnostic: crictl inspect <CID>.

5. Dependency not ready : Exit code 1 / Error. Trigger: Start-up race: can't reach DB/Redis. Key diagnostic: Check initContainers.

6. Ephemeral storage full : Exit code 137 / Evicted. Trigger: Logs/temp files exceed ephemeral-storage limit. Key diagnostic: df -h /var/lib/containerd.

7. Node kernel resource exhaustion : Exit code 1 or 134 / Error. Trigger: pid_max / inotify / nofile limits hit. Key diagnostic: sysctl -a.

8. Graceful termination timeout : Exit code 137 / Error. Trigger: Process ignores SIGTERM, gets SIGKILL. Key diagnostic: Check preStop and PID 1.

1.3 Exit Code ↔ Signal Mapping & Backoff State Machine

Linux/Kubernetes follow 128 + Signal POSIX spec. Key mappings: 1 / 2: App generic error. Common trigger: Missing config, connection timeout, syntax error. Triage: First kubectl logs --previous. 126 / 127: Not executable / command not found. Common trigger: Script lacks +x or PATH wrong. Triage: Use kubectl debug to inspect image. 134: SIGABRT (128+6). Common trigger: glibc assert, Go/C++ panic. Triage: Check core/stderr & system lib versions. 137: SIGKILL (128+9). Common trigger: Kernel OOMKilled or probe kill. Triage: Check dmesg & describe Events. 139: SIGSEGV (128+11). Common trigger: Illegal memory access / null pointer. Triage: Verify arch match ( amd64 / arm64). 143: SIGTERM (128+15). Common trigger: Kubelet probe failure or rollout. Triage: Check livenessProbe & preStop.

1.4 Deep Dive: Each Cause

Cgroup Memory Limit (OOMKilled / Exit 137) : When anon-rss + page cache hits memory.max, kernel OOM Killer sends SIGKILL. Common with Java off-heap leaks, Python data loading, unbounded goroutines.

Startup Config/Secret Missing or App Bug (Exit 1/2) : ConfigMap key typo, Secret not synced, YAML indent error, bad DB URL → uncaught exception in main() init.

LivenessProbe Misconfiguration (Exit 143/137) : No startupProbe; cold start exceeds initialDelaySeconds + periodSeconds * failureThreshold → livenessProbe kills before ready.

Image Arch Mismatch or Missing Exec Perm (Exit 126/127/255) : ARM64 node pulls linux/amd64 -only image → exec format error; shell script lacks chmod +x or has Windows CRLF (

).

Dependency Race (Exit 1) : Business Pod starts before MySQL/Redis/Kafka/Nacos; connection pool init fails without retry backoff.

Ephemeral Storage Full (Exit 137 / Evicted) : No persistent volume; large logs/temp files/model cache written to /tmp or container writable layer → exceeds limits.ephemeral-storage or triggers node DiskPressure eviction.

Node Kernel Quota Exhaustion (Exit 1/134) : Host fs.inotify.max_user_instances, kernel.pid_max, vm.max_map_count hit ceiling → pthread_create failed or config watch fails.

PID 1 Zombie & Graceful Exit Timeout (Exit 137) : sh -c "java ..." without exec makes shell PID 1; it doesn't forward SIGTERM to child. Kubelet waits terminationGracePeriodSeconds then sends SIGKILL, risking data corruption on next start.

1.5 AI/Big Data Hidden Trap: /dev/shm 64 MB Default

PyTorch, vLLM, multi-process Python inference containers get only 64 MB shared memory by default. DataLoader or NCCL shared-memory transport instantly fills it → SIGBUS (Exit 135) or Bus error → CrashLoopBackOff. Fix: mount memory-backed emptyDir over /dev/shm:

volumeMounts:
  - mountPath: /dev/shm
    name: dshm
volumes:
  - name: dshm
    emptyDir:
      medium: Memory
      sizeLimit: 8Gi

Note : medium: Memory counts against limits.memory; raise limit to include the 8 GiB shm budget or OOMKilled (137) recurs.

2. Dual-Layer Hardening: Pod Spec + Node/OS/Kubelet/Containerd

2.1 Application-Layer: Three-Probe Separation & Guaranteed QoS Template

Over 40% of CrashLoopBackOff stems from conflating startupProbe with livenessProbe or missing initContainers. Production-validated Deployment:

# prod-order-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: prod-core
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      terminationGracePeriodSeconds: 45
      # 1. Dependency wait: eliminates start-up race (cause #5)
      initContainers:
        - name: wait-for-mysql
          image: busybox:1.36.1
          command: ["sh", "-c", "until nc -z -w 2 mysql-primary.prod-db.svc.cluster.local 3306; do echo 'Waiting for MySQL 3306...'; sleep 2; done"]
          resources:
            requests: {cpu: "50m", memory: "64Mi"}
            limits: {cpu: "100m", memory: "64Mi"}
      containers:
        - name: order-service
          image: registry.corp.internal/core/order-service:v2.4.1
          envFrom:
            - configMapRef: {name: order-service-config}
          env:
            - name: DB_PASSWORD
              valueFrom: {secretKeyRef: {name: order-db-secret, key: password}}
            # Reserve 25% off-heap for Metaspace, DirectMemory, thread stacks
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -Xms3072m -Xmx3072m
                -XX:MaxMetaspaceSize=512m
                -XX:MaxDirectMemorySize=256m
                -XX:+UseG1GC
                -XX:+ExitOnOutOfMemoryError
                -XX:+HeapDumpOnOutOfMemoryError
                -XX:HeapDumpPath=/var/log/app/heapdump.hprof
          resources:
            requests: {cpu: "2", memory: "4Gi", ephemeral-storage: "2Gi"}
            limits: {cpu: "2", memory: "4Gi", ephemeral-storage: "4Gi"}
          # 4. Three-probe separation
          startupProbe:
            httpGet: {path: /actuator/health/liveness, port: 8080}
            failureThreshold: 30
            periodSeconds: 5
            timeoutSeconds: 3
          readinessProbe:
            httpGet: {path: /actuator/health/readiness, port: 8080}
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          livenessProbe:
            httpGet: {path: /actuator/health/liveness, port: 8080}
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3
          # 5. Graceful drain
          lifecycle:
            preStop:
              exec: {command: ["sh", "-c", "sleep 10"]}
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 10001
          volumeMounts:
            - name: app-logs
              mountPath: /var/log/app
      volumes:
        - name: app-logs
          emptyDir: {sizeLimit: 2Gi}

2.1.1 StatefulSet Trap: PVC Lock & WAL Corruption

StatefulSet Pods (DB, MQ) often CrashLoopBackOff due to stuck RWO volume or stale lock files ( postmaster.pid, LOCK). Safe pre-check in initContainers:

initContainers:
  - name: check-stale-lock
    image: busybox:1.36.1
    command:
      - sh
      - -c
      - |
        if [ -f /var/lib/postgresql/data/postmaster.pid ]; then
          echo "[WARN] Stale postmaster.pid detected, manual check required"
          cat /var/lib/postgresql/data/postmaster.pid
        fi
    volumeMounts:
      - name: pgdata
        mountPath: /var/lib/postgresql/data

2.2 Node OS Kernel, Containerd & Kubelet Layer

Even perfect YAML fails if node kernel params ( inotify, pid_max) or Kubelet eviction thresholds are untuned. Required /etc/sysctl.d/99-k8s-node.conf:

# /etc/sysctl.d/99-k8s-node.conf
fs.inotify.max_user_watches = 1048576
fs.inotify.max_user_instances = 8192
kernel.pid_max = 4194304
kernel.threads-max = 2097152
vm.max_map_count = 262144
vm.swappiness = 0
vm.overcommit_memory = 1

Apply: sysctl --system (hot, no disruption).

Kubelet /var/lib/kubelet/config.yaml stability & log rotation:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
containerLogMaxSize: "100Mi"
containerLogMaxFiles: 5
systemReserved:
  cpu: "500m"
  memory: "1Gi"
  ephemeral-storage: "5Gi"
kubeReserved:
  cpu: "500m"
  memory: "1Gi"
  ephemeral-storage: "5Gi"
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
podPidsLimit: 4096

Restart: systemctl restart kubelet (control-plane heartbeat blip ~seconds, running containers unaffected).

Containerd /etc/containerd/config.toml must match Kubelet Cgroup driver:

version = 2
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Restart: systemctl restart containerd (briefly affects new Pod creation; existing systemd-cgroup containers survive).

2.2.1 Image Build & PID 1 Signal Forwarding

To eliminate cause #4 (perm/CRLF) and #8 (PID 1 swallows SIGTERM), use tini as init and enforce perms at build:

FROM eclipse-temurin:21.0.4_7-jre-jammy
RUN apt-get update && apt-get install -y --no-install-recommends tini curl && \
    rm -rf /var/lib/apt/lists/* && \
    groupadd -g 10001 appgroup && useradd -u 10001 -g appgroup -s /bin/false appuser
WORKDIR /app
COPY --chown=appuser:appgroup target/order-service.jar /app/order-service.jar
COPY --chown=appuser:appgroup scripts/entrypoint.sh /app/entrypoint.sh
RUN sed -i 's/\r$//' /app/entrypoint.sh && chmod 0755 /app/entrypoint.sh
USER 10001:10001
ENTRYPOINT ["/usr/bin/tini", "--", "/app/entrypoint.sh"]

Entrypoint must exec the JVM so it becomes PID 1 (or child of tini) and receives signals:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'
\t'
echo "[INFO] Starting order-service with JAVA_TOOL_OPTIONS=${JAVA_TOOL_OPTIONS:-none}"
exec java -jar /app/order-service.jar

2.2.2 Kubelet Image & Ephemeral Storage GC

Without auto-cleanup, /var/lib/containerd >85% triggers DiskPressure → mass eviction + CrashLoopBackOff. Add to Kubelet config:

imageGCHighThresholdPercent: 80
imageGCLowThresholdPercent: 65
imageMinimumGCAge: 2m0s
evictionSoft:
  nodefs.available: "15%"
  imagefs.available: "20%"
evictionSoftGracePeriod:
  nodefs.available: "1m30s"
  imagefs.available: "1m30s"

2.2.3 Graceful Shutdown & SIGTERM Chain Timing

Cause #8 often happens during rollout/node drain: container ignores SIGTERM, gets SIGKILL, leaves WAL/lock files → next start crashes. Standard sequence:

[K8s 触发 Pod 删除]
     |
     +---> [Endpoint Controller 摘除 Pod IP] (async 1~5s)
     |
     +---> [Kubelet 执行 preStop: sleep 10] (等待流量摘除)
               | 
               v (10s 后)
      [Kubelet 发送 SIGTERM 给 PID 1]
               | 
               v (tini 转发给业务主进程)
      [应用停止接收新请求并排空连接池] (耗时 10~25s)
               | 
               v (总耗时 < 45s)
      [进程返回 Exit 0 正常退出]

Nginx: on SIGTERM drops connections; use preStop with nginx -s quit loop. Python Uvicorn: must be PID 1 (or under tini) with --timeout-graceful-shutdown.

2.2.4 Native Sidecar (K8s v1.37 GA) Eliminates Mesh/Log Sidecar Race

Traditional parallel start: main container calls deps before Envoy ready → connection refused → CrashLoopBackOff. Native sidecar in initContainers with restartPolicy: Always:

initContainers:
  - name: envoy-proxy
    image: envoyproxy/envoy:v1.31.2
    restartPolicy: Always
    ports:
      - containerPort: 15001
        name: proxy-outbound
    startupProbe:
      httpGet: {path: /ready, port: 15000}
      periodSeconds: 2
      failureThreshold: 15
    resources:
      requests: {cpu: "200m", memory: "256Mi"}
      limits: {cpu: "500m", memory: "512Mi"}

Kubelet waits for sidecar startupProbe success before starting main; on teardown, waits for main to exit before stopping sidecar.

2.3 Four-Dimensional Config Breakdown

① Must-change per cluster : namespace, image registry, initContainers service DNS/ports, Secret/ConfigMap refs.

② Coupled dependencies : cgroupDriver ↔ SystemdCgroup alignment; HeapDumpPath ↔ emptyDir.sizeLimit (set ≥ Xmx * 1.2); terminationGracePeriodSeconds > preStop sleep + app drain time (45s example).

③ Formulas not hard numbers : Java Xmx = Limit * 0.70~0.75 (e.g., 4 GiB limit → 3072 MiB heap, 1024 MiB for Metaspace/DirectMemory/thread stacks/JVM overhead). StartupProbe max = failureThreshold * periodSeconds (30*5s=150s).

④ Red lines : Never point livenessProbe & readinessProbe at same downstream dependency (DB blip → cluster-wide kill). Never set podPidsLimit: -1 (fork bomb kills node). Never drop systemReserved (OOM Killer may kill kubelet/containerd).

2.4 Config Apply & Baseline Verification

sysctl --system | grep -E "max_user_watches|pid_max|vm.max_map_count"
kubectl -n prod-core rollout status deployment/order-service
kubectl -n prod-core get pod -l app=order-service -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.qosClass}{"
"}{end}'

Expected: sysctl values applied, rollout succeeds, all Pods show Guaranteed QoS.

3. High-Frequency One-Liners & PromQL Alerts

3.1 Rapid Root-Cause Extraction

Pod in CrashLoopBackOff, container stopped; kubectl logs empty. Use --previous for last crash stdout; jsonpath for exitCode, reason, finishedAt:

POD_NAME="order-service-7b9d4c8f1-k9x2p" && \
kubectl -n prod-core get pod "${POD_NAME}" -o jsonpath='{range .status.containerStatuses[*]}{"Container: "}{.name}{"
Restarts: "}{.restartCount}{"
LastReason: "}{.lastState.terminated.reason}{"
ExitCode: "}{.lastState.terminated.exitCode}{"
Message: "}{.lastState.terminated.message}{"
"}{end}'

Output example:

Container: order-service
Restarts: 7
LastReason: OOMKilled
ExitCode: 137
Message:

Then pull last 80 lines of previous log + Events:

kubectl -n prod-core logs "${POD_NAME}" --previous --tail=80
kubectl -n prod-core describe pod "${POD_NAME}" | sed -n '/Events:/,$p'

Events reveal probe failures & Kubelet killing:

Warning  Unhealthy  11m (x3 over 24m)  kubelet  Liveness probe failed: HTTP probe failed with statuscode: 503
Normal   Killing    11m (x3 over 24m)  kubelet  Container order-service failed liveness probe, will be restarted
Warning  BackOff    2m (x42 over 22m)  kubelet  Back-off restarting failed container

3.1.1 Multi-Replica Log Aggregation ( stern )

stern -n prod-core -l app=order-service --tail=30 --include "Exception|Error|FATAL|OOM"

3.2 kubectl debug Zero-Restart Diagnostic Container

Container crashes in 1s ( exec fails). kubectl debug creates a copy Pod sharing FS/namespaces, overrides command to sleep 3600:

kubectl -n prod-core debug "${POD_NAME}" -it --copy-to=debug-order-pod --container=order-service -- sh

Or ephemeral busybox for network/fs checks:

kubectl -n prod-core debug -it "${POD_NAME}" --image=busybox:1.36.1 --target=order-service

3.2.1 crictl Inspect Crashed Container FS State

On worker node, find exited container ID, inspect config/env/mounts:

CID=$(crictl ps -a --name order-service -q | head -n 1) && crictl inspect "${CID}" | jq '{exitCode: .status.exitCode, reason: .status.reason, cmd: .info.config.command, args: .info.config.args, mounts: [.info.config.mounts[].container_path]}'

Output:

{
  "exitCode": 127,
  "reason": "Error",
  "cmd": ["/app/start.sh"],
  "args": null,
  "mounts": ["/var/log/app", "/etc/config"]
}

Check raw container log file:

LOG_PATH=$(crictl inspect "${CID}" | jq -r '.status.logPath') && tail -n 20 "${LOG_PATH}"

Reveals: /bin/sh: 1: /app/start.sh: not found (path wrong or missing shebang).

3.3 Host crictl + dmesg Cross-Verification

API shows ExitCode 137/255 but Events lack OOMKilled/Liveness. SSH to worker node:

NODE_IP=$(kubectl -n prod-core get pod "${POD_NAME}" -o jsonpath='{.status.hostIP}')
crictl ps -a --name order-service
crictl inspect "$(crictl ps -a --name order-service -q | head -n 1)" | jq '.status | {exitCode, reason, startedAt, finishedAt}'
dmesg -T | grep -iE "oom-killer|killed process|cgroup" | tail -n 10

Kernel log confirms:

[Fri Nov 13 10:14:22 2026] Memory cgroup out of memory: Killed process 48291 (java) total-vm:6142800kB, anon-rss:4128900kB, file-rss:12400kB, shmem-rss:0kB, UID:10001

3.4 Cluster-Wide CrashLoopBackOff Distribution

kubectl get pods -A --field-selector=status.phase!=Succeeded -o json | \
jq -r '.items[] | select(.status.containerStatuses[]?.state.waiting.reason == "CrashLoopBackOff") | [.metadata.namespace, .metadata.name, (.status.containerStatuses[0].restartCount|tostring), (.status.containerStatuses[0].lastState.terminated.reason // "Unknown"), (.status.containerStatuses[0].lastState.terminated.exitCode|tostring)] | @tsv' | column -t

Output:

prod-core      order-service-7b9d4c8f1-k9x2p   14  OOMKilled  137
ai-inference   vllm-worker-6f8a2b1c-x9m2p     8  Error      1
logging        fluent-bit-ds-9k2m1           22  Error      134

3.4.1 Ephemeral Storage & Inode Exhaustion

df -h /var/lib/containerd /var/log && df -i /var/lib/containerd
du -h --max-depth=2 /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots 2>/dev/null | sort -hr | head -n 10

Shows 92% disk / 98% inodes used; 18 GB in one snapshot → container writing huge temp files without volume.

3.4.2 Image Arch & ConfigMap/Secret Pre-Check

kubectl -n prod-core get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture,OS:.status.nodeInfo.osImage
kubectl -n prod-core get cm,secret | grep -E "order-service-config|order-db-secret"

Reveals mixed arch cluster (amd64 + arm64) but image only amd64 → Exit 255 on arm64 nodes.

3.4.3 Host Cgroup Real-Time & Pids/Shm Inspection

kubectl -n prod-core exec "${POD_NAME}" -- sh -c 'df -h /dev/shm && ulimit -n -u'

Shows shm 64M 100% → PyTorch/Chromium will Bus error. Check Kubelet journal for eviction/kill timeline:

journalctl -u kubelet --since "30 minutes ago" --no-pager | grep -E "Killing container|OOM|Evicted|FailedMount" | tail -n 15

3.4.4 CoreDNS & NetworkPolicy Diagnosis

kubectl -n prod-core run tmp-netcheck --rm -i --tty --restart=Never --image=nicolaka/netshoot:v0.13 -- sh -c 'nslookup mysql-primary.prod-db.svc.cluster.local && nc -zv -w 3 mysql-primary.prod-db.svc.cluster.local 3306'

DNS resolves but TCP 3306 times out → NetworkPolicy/Egress block. Check: kubectl -n prod-core get networkpolicy -o wide Shows deny-all-egress selecting app=order-service without 3306 allow.

3.5 Production PromQL Alert Rules

groups:
  - name: kubernetes-pod-stability
    rules:
      - alert: KubePodCrashLooping
        expr: max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) >= 1
        for: 3m
        labels: {severity: critical}
        annotations:
          summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} in CrashLoopBackOff"
          description: "Container {{ $labels.container }} restart backoff > 3 min"
      - alert: KubePodOOMKilled
        expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 and increase(kube_pod_container_status_restarts_total[10m]) > 0
        for: 1m
        labels: {severity: critical}
      - alert: KubePodRestartRateHigh
        expr: increase(kube_pod_container_status_restarts_total[15m]) >= 3
        for: 2m
        labels: {severity: warning}
      - alert: KubePodEvictedDiskOrMemory
        expr: kube_pod_status_reason{reason="Evicted"} == 1
        for: 1m
        labels: {severity: warning}
      - alert: NodeInodeExhaustionWarning
        expr: (1 - (node_filesystem_files_free{mountpoint="/var/lib/containerd"} / node_filesystem_files{mountpoint="/var/lib/containerd"})) * 100 > 85
        for: 5m
        labels: {severity: critical}

4. Three Deep-Dive Real-World Fault Loops

4.1 Loop 1: Java Heap Not Full Yet Frequent OOMKilled (Off-Heap Leak)

Symptom : Payment reconciliation service ( limits.memory: 4Gi, JVM -Xms3584m -Xmx3584m) OOMKilled nightly; Grafana shows Old Gen only 62%.

Evidence : Cgroup v2 memory.events shows oom_kill 1. jcmd 1 VM.native_memory summary reveals committed 4184 MB > 4096 MB limit. Breakdown: Heap 3584 MB + Thread 248 MB + Internal (Netty DirectByteBuffer) 312 MB + GC 184 MB + Class 142 MB = 4470 MB. memory.stat anon 3.84 GB (non-reclaimable).

Root Cause : Xmx at 87.5% of limit left only 512 MB for off-heap; actual off-heap > 886 MB → kernel OOMKilled before JVM throws OOME.

Fix : Xmx=2800m (70%), explicit -XX:MaxDirectMemorySize=384m, -XX:MaxMetaspaceSize=384m, raise limit to 5 GiB.

Before/After : Peak RSS 4096 MB (killed) → 3620 MB (1.4 GB headroom); 11 OOMKilled/24h → 0; batch job from repeated failures → 4m18s single run.

4.2 Loop 2: High-Throughput/Slow-Init Service Repeatedly Murdered by LivenessProbe

Symptom : AI search service loads 2.4 GB vector index (~75s) on start; all new Pods die at ~45s with Exit 143 ( SIGTERM).

Evidence : describe pod shows livenessProbe only:

initialDelaySeconds: 15, periodSeconds: 10, failureThreshold: 3

→ max 45s. Events: Liveness probe failed: connection refused → Killing at 45s.

Root Cause : No startupProbe; 75s init exceeds 45s liveness window.

Fix : Add startupProbe ( periodSeconds: 5, failureThreshold: 30 → 150s grace); liveness focuses on process health.

Before/After : Startup ceiling 45s (certain death) → 150s (78s ready); rollout failure 100% → 0%.

4.3 Loop 3: Node inotify Exhaustion Triggers Multi-Pod Collective CrashLoopBackOff

Symptom : 6 Pods (Nginx, FluentBit, SpringBoot) on one dense node (68 Pods) simultaneously CrashLoopBackOff; fine on other nodes.

Evidence : FluentBit --previous log: inotify_init failed: Too many open files. Node sysctl fs.inotify.max_user_instances = 128 (default). find /proc/*/fd -lname anon_inode:inotify shows top processes holding 64, 48, 16 instances.

Root Cause : All containers share host kernel inotify quota per UID; log shippers + Java FileWatchers consume all 128 instances → new Pods fail inotify_init() with EMFILE.

Fix : /etc/sysctl.d/99-k8s-node.conf set max_user_instances=8192, max_user_watches=1048576; sysctl --system; delete backoff Pods to force rebuild.

Before/After : Instances 128 (100% used) → 8192 (<2% used); 6 CrashLoopBackOff → 0 (all Running).

4.4 Supplemental: Image Arch Mismatch & CRLF Script Exit (127/255)

Go sync service scheduled onto new ARM64 nodes; exec format error. Node arch arm64, image only amd64. Short-term: nodeSelector: kubernetes.io/arch: amd64; long-term: docker buildx build --platform linux/amd64,linux/arm64.

5. Automation Scripts, Risk Matrix & Engineering Iron Laws

5.1 10-Second Auto Triage Script ( k8s_crash_triage.sh )

Extracts exitCode, Events, previous logs, node info; includes --dry-run; auto-diagnoses based on exitCode (137→OOM/probe, 143→liveness, 126/127/255→perm/arch, 1/2→config/logs).

5.2 Node-Level OOM & Cgroup Audit Script ( k8s_node_oom_check.sh )

Read-only check of sysctl inotify/pid_max, dmesg OOM records, containerd disk usage.

5.3 Production Change Risk Matrix

Change : Deployment probe/limit. Impact Window : Pod rolling rebuild; off-peak. Pre-check/Backup : kubectl get deploy -o yaml > bak.yaml. Canary & Rollback : maxUnavailable: 0; rollout undo.

Change : kubelet/config.yaml. Impact Window : Kubelet restart ~seconds disconnect. Pre-check/Backup : Backup original config.yaml. Canary & Rollback : Single-node canary; restore file & restart.

Change : containerd/config.toml. Impact Window : Runtime restart affects new Pods. Pre-check/Backup : cp config.toml config.toml.bak. Canary & Rollback : drain node; restore & restart.

Change : sysctl --system. Impact Window : Global kernel immediate. Pre-check/Backup : sysctl -a > /tmp/sysctl.bak. Canary & Rollback : Per-node observe 5m; sysctl -p rollback.

5.3.1 30-Second CrashLoopBackOff Decision Tree

[发现 Pod 状态 CrashLoopBackOff]
         |
         v
 [提取 lastState.terminated.exitCode]
         |
  +-----------+-----------+-----------+
  v           v           v           v
[Exit 137]  [Exit 143]  [Exit 1/2]  [126/127/255]
  |           |           |           |
  v           v           v           v
查 dmesg    查 describe  查 logs     查 kubectl debug
OOM/Evict?  Liveness?   --previous  权限/架构/CRLF
  |           |           |           |
  v           v           v           v
调 Xmx 75%  加 startup  补 Config/  改 +x / exec /
或扩 Limit  探针宽容期  Init等待    multi-arch镜像

5.3.2 Standardized Fix & Prevention Checklist

Cause 1. OOMKilled : Immediate: Raise limits.memory. Long-term: Lock Xmx at Limit 75%. Verify: jcmd 1 VM.native_memory.

Cause 2. Config/Secret : Immediate: Rollback ConfigMap/Secret. Long-term: CI YAML Schema validation. Verify: kubectl logs --previous.

Cause 3. Probe Mis-kill : Immediate: Temp increase initialDelay. Long-term: Split startupProbe from liveness. Verify: kubectl describe pod.

Cause 4. Perm/Arch : Immediate: Pin nodeSelector arch. Long-term: Build multi-arch + +x. Verify: kubectl debug.

Cause 5. Dependency : Immediate: Manual restart after ready. Long-term: Add initContainers probes. Verify: kubectl get pod -w.

Cause 6. Ephemeral Full : Immediate: Clean node zombie images. Long-term: Set emptyDir.sizeLimit. Verify: df -h /var/lib/containerd.

Cause 7. Kernel Quota : Immediate: sysctl -w temp raise. Long-term: Harden /etc/sysctl.d/ baseline. Verify: sysctl -a.

Cause 8. Exit Timeout : Immediate: Extend gracePeriod. Long-term: Adopt tini + exec launch. Verify: journalctl -u kubelet.

5.3.3 Extra High-Frequency kubectl Snippets

Multi-container Pod: iterate all containers for previous logs:

for c in $(kubectl -n prod-core get pod "${POD_NAME}" -o jsonpath='{.spec.containers[*].name}'); do
  echo ">>> Container: ${c} <<<"
  kubectl -n prod-core logs "${POD_NAME}" -c "${c}" --previous --tail=30 2>&1 || true
done

Recent cluster-wide Warning events by timestamp:

kubectl -n prod-core get events --field-selector type=Warning --sort-by='.lastTimestamp' | tail -n 20

5.3.4 Persistent HeapDump Volume for OOM Forensics

Instead of losing .hprof on Pod recreate, mount hostPath or PVC:

volumeMounts:
  - name: heapdump-vol
    mountPath: /var/log/heapdump
volumes:
  - name: heapdump-vol
    hostPath:
      path: /data/k8s-heapdumps
      type: DirectoryOrCreate

Clean expired dumps to avoid reverse DiskPressure.

5.3.5 Safe Evicted Pod Cleanup

kubectl get pods -A --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason == "Evicted") | "\(.metadata.namespace) \(.metadata.name)"' | while read -r ns name; do
  echo "Deleting Evicted Pod: ${ns}/${name}"
  kubectl -n "${ns}" delete pod "${name}"
done

Pre-check per-node eviction count:

kubectl get pods -A --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason == "Evicted") | .spec.nodeName' | sort | uniq -c | sort -nr

5.4 Four SRE Iron Laws for CrashLoopBackOff

Exit code trumps surface status : First extract lastState.terminated.exitCode; 137→OOM/kill, 143→probe SIGTERM, 126/127→image cmd/arch. Never guess.

Three probes, distinct roles : Cold start → startupProbe; traffic drain → readinessProbe; process deadlock → livenessProbe. livenessProbe must never hard-depend on external DB.

Container memory limit must reserve 25% for off-heap : JVM/Python/Go Xmx or model cache ≤ 70-75% of limits.memory; prevents kernel OOMKilled before app-level OOME.

Grab --previous & dmesg before deleting Pod : CrashLoopBackOff is self-protection; blind delete pod destroys forensic evidence.