Kubernetes ‘Deadlock’ Explained: Guide to Diagnosing and Fixing Performance Issues

Incident Background

A Kubernetes 1.28 cluster with ~200 nodes, 50+ micro‑services, Istio sidecars, Nacos, Kafka and a Prometheus‑Grafana‑Loki stack was load‑tested before a promotion. After 12 minutes the cluster showed a classic “dead‑lock” symptom: order‑service P99 latency rose from 60ms to 2.5s, throughput dropped, error rate surged, some pods were OOMKilled, HPA oscillated between 10‑60 replicas, node‑level CPU throttling appeared despite low host CPU, kubectl get pods -A became extremely slow, CoreDNS timeouts exploded and a few nodes entered NotReady, triggering cascading evictions. Node‑average CPU stayed at 40‑50 % and memory/bandwidth were not saturated, indicating a feedback‑loop of localized bottlenecks rather than a resource‑full cluster.

Correct Mental Model: Cascading Block

Under massive load, Kubernetes behaves as an asynchronous, weakly consistent system; a small jitter can amplify into a system‑wide congestion loop:

Load ↑ → application concurrency ↑ → CPU throttling / memory reclamation / thread pile‑up → request timeouts & retries → DNS query explosion → short‑connection surge → nf_conntrack table growth → Service forwarding jitter → more timeouts & retries → pod health‑check failures → pod restarts & HPA thrashing → API‑server & etcd pressure → control‑plane latency ↑ → node heartbeats blocked → cluster appears NotReady.

This positive‑feedback chain spans four often‑ignored layers:

Container resource control plane : CPU CFS limits, memory working‑set, GC spikes, thread‑pool mismatches.

Node kernel network plane : nf_conntrack, iptables/IPVS/eBPF, socket backlog, TIME_WAIT, port exhaustion.

Kubernetes control plane : API‑server QPS, API Priority & Fairness (APF), informer usage, etcd latency.

Infrastructure amplifiers : CoreDNS, service‑mesh sidecars, retry mechanisms, circuit‑breaker mis‑configurations, HPA tuning.

Step‑by‑Step Post‑Mortem

Phase 1 – Discard Physical‑Resource‑Full Illusion

Business metrics (gateway success rate, order‑service P99, Kafka lag) degraded while node panels showed CPU < 50 %, memory far from limit and no bandwidth saturation. The problem was “resource usage blocked”, not “resource exhausted”.

Phase 2 – Detect CPU Throttling & OOMKill Co‑existence

Prometheus queries revealed persistent spikes in container_cpu_cfs_throttled_seconds_total and memory working‑set approaching the container limit, causing frequent Full GC and OOMKill. Containers were both CPU‑throttled and memory‑starved, inflating latency.

Phase 3 – Network & Control‑Plane as the Second Trigger

Application slowdown caused a cascade of retries, short connections and DNS queries. nf_conntrack entries neared the kernel limit ( cat /proc/sys/net/netfilter/nf_conntrack_max, conntrack -C), leading to packet drops. API‑server request queues and flow‑control rejections grew, confirming control‑plane participation.

Core Root Causes

CPU Usage ≠ CPU Availability

Kubernetes maps limits.cpu to Linux CFS bandwidth control ( cpu.cfs_period_us =100 ms). A pod with limits.cpu=2 can consume at most 200 ms of CPU per period; high‑concurrency workloads can exhaust this quota within a single period, causing the kernel to pause the container even though the host shows idle CPU. Therefore both container_cpu_usage_seconds_total and container_cpu_cfs_throttled_seconds_total must be inspected.

Services Most Sensitive to CPU Limits

Java web services with large thread pools and aggressive GC.

Go services under high goroutine concurrency.

Node.js single‑threaded event loops.

Sidecar proxies (Envoy, Nginx) sharing the same cgroup quota.

Memory Working‑Set vs. Heap

Container memory limits cap the total budget, which includes heap, metaspace, direct memory, thread stacks, code cache, Netty buffers, mmap files and sidecar memory. A typical mis‑configuration is limit=4Gi with JVM -Xmx3g; the remaining 1 Gi is quickly consumed by non‑heap usage, leading to OOMKill under load.

Recommended Memory Budget Split

Heap: 50‑65 % of container limit.

Non‑heap + native: 15‑25 %.

Burst buffer: 10‑15 %.

Sidecar: allocate based on observed load.

Conntrack Saturation

Kubernetes services, SNAT/DNAT rely on nf_conntrack. Massive short‑connection traffic can fill the conntrack table, causing packet drops, retransmits and timeouts. The following commands help diagnose the condition:

cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count
conntrack -S
ss -s

Prometheus metric: node_nf_conntrack_entries / node_nf_conntrack_entries_limit. Alert thresholds: >70 % (warning), >85 % (high risk), >95 % (critical).

iptables‑mode kube‑proxy & Service Mesh Amplification

In iptables mode each Service/Endpoint adds iptables rules; scaling services and replicas quickly inflates rule count, making updates expensive. During the incident frequent pod restarts and HPA scaling caused massive re‑programming latency. Metric:

histogram_quantile(0.99, sum by (le,instance) (rate(kube_proxy_network_programming_duration_seconds_bucket[5m])))

P99 > 3‑5 s indicates a Service‑programming bottleneck.

Mitigations

Switch kube‑proxy to IPVS.

Limit sidecar injection to critical paths.

Reduce Service‑layer hops.

CoreDNS as a Symptom Amplifier

High query volume, SDKs without caching, sidecar repeated lookups and the default ndots:5 cause many unnecessary internal lookups, inflating DNS latency. Mitigations:

Lower ndots to 2 for critical services.

Deploy NodeLocal DNSCache.

Scale CoreDNS (e.g., 6 replicas) and enable HPA.

Control‑Plane Overload

API‑server request latency and flow‑control rejections grew as pod restarts, HPA queries and endpoint churn increased. APF can throttle low‑priority requests, making kubectl get pods appear hung. Metrics: apiserver_request_duration_seconds (P99). apiserver_flowcontrol_rejected_requests_total.

etcd latency spikes.

Comprehensive Remediation Framework

Production‑Grade Metrics to Monitor

CPU throttling ratio:

sum(rate(container_cpu_cfs_throttled_periods_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

CPU usage:

sum(rate(container_cpu_usage_seconds_total[1m])) by (pod)

Memory working‑set / limit:

sum(container_memory_working_set_bytes) by (pod) / sum(kube_pod_container_resource_limits{resource="memory",unit="byte"}) by (pod)

Conntrack usage:

node_nf_conntrack_entries / node_nf_conntrack_entries_limit

CoreDNS P99 latency, NXDOMAIN/SERVFAIL rates.

Kube‑proxy programming latency (same metric as above).

API‑server queue latency and rejected‑request count.

Business P95/P99 + error rate correlation with throttling and GC pauses.

Recommended Alert Thresholds

Throttling ratio > 5 % → investigate CPU limits.

Core service throttling > 15 % → expect P99 impact.

Conntrack > 70 % (warning), > 85 % (high risk), > 95 % (critical).

API‑server P99 > 3 s → suspect control‑plane overload.

Sample Manifests

CPU‑limit removal for latency‑sensitive services :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: trade
spec:
  replicas: 12
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: app
        image: registry.example.com/trade/order-service:2026.04.28
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "1500m"
            memory: "2Gi"
          limits:
            memory: "2Gi"
        env:
        - name: JAVA_TOOL_OPTIONS
          value: |
            -XX:+UseContainerSupport
            -XX:InitialRAMPercentage=50
            -XX:MaxRAMPercentage=70
            -XX:+AlwaysActAsServerClassMachine
            -XX:+HeapDumpOnOutOfMemoryError
            -XX:HeapDumpPath=/dumps
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          periodSeconds: 5
          timeoutSeconds: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10

NodeLocal DNSCache DaemonSet :

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: node-local-dns
  template:
    metadata:
      labels:
        app: node-local-dns
    spec:
      dnsPolicy: Default
      hostNetwork: true
      containers:
      - name: node-cache
        image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.28
        args:
        - -localip
        - 169.254.20.10

IPVS kube‑proxy via Cilium :

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=${API_SERVER_IP} \
  --set k8sServicePort=6443 \
  --set ipam.mode=kubernetes \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Runbook (5‑minute actions)

Pause the load test or cut incoming traffic.

Stop non‑essential batch jobs, data back‑fills and offline tasks.

Scale up core‑service replicas and freeze non‑critical releases.

Temporarily relax retry policies or downgrade read‑only endpoints.

If conntrack > 90 %, add nodes or migrate hot load to a separate pool.

15‑Minute KPI Checklist

CPU throttling ratio for core pods.

OOMKill events and Full GC duration spikes.

Conntrack usage percentage.

CoreDNS P99 latency.

API‑server request queue length and rejected‑request count.

Kube‑proxy or CNI error logs.

Post‑Incident Governance Tasks

Standardise service resource templates; avoid blind copy‑paste.

Build latency‑centric capacity baselines (not just average CPU).

Adopt PDB, TopologySpread, and node‑pool isolation for critical namespaces.

Enforce platform‑wide limits on retries, timeouts, and connection‑pool sizes.

Run pre‑promotion drills for DNS, conntrack, and control‑plane stability.

Structured Troubleshooting Checklist

Business vs. Platform Correlation

Business error rate, P95/P99, gateway timeouts vs. platform metrics.

Container‑cgroup Health

container_cpu_cfs_throttled_seconds_total

container_memory_working_set_bytes

OOMKill events, GC pause times.

Node Network Congestion

Conntrack usage ratio. ss -s output, TCP retransmits.

Kube‑proxy programming latency.

DNS Amplification

CoreDNS P99 latency, NXDOMAIN / SERVFAIL rates.

Current ndots setting.

NodeLocal DNSCache deployment status.

Control‑Plane Saturation

API‑server request duration histogram.

Flow‑control rejected requests.

etcd latency.

Abnormal List/Watch patterns from custom controllers.

Business Governance Review

Timeout budgets vs. retry budgets.

Circuit‑breaker placement.

HPA stability (scale‑up/down windows, metrics).

Node‑pool and service isolation.

Final Takeaway

Kubernetes is not merely a resource orchestrator; it is a stack of Linux kernel controls, distributed networking, massive state orchestration and service‑mesh amplification. “Deadlock” arises from multiple subsystems entering low‑efficiency, queued and retry‑driven states simultaneously. Preventing it requires system‑wide constraints, right feedback‑loop metrics and graceful degradation design.

PromQL Metrics Reference List

# CPU throttling ratio
sum(rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])) by (pod) /
sum(rate(container_cpu_cfs_periods_total{container!=""}[5m])) by (pod)

# Memory working set / limit
sum(container_memory_working_set_bytes{container!=""}) by (pod) /
sum(kube_pod_container_resource_limits{resource="memory",unit="byte"}) by (pod)

# Conntrack usage
node_nf_conntrack_entries / node_nf_conntrack_entries_limit

# CoreDNS P99 latency
histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m])))

# Kube‑proxy programming latency
histogram_quantile(0.99, sum by (le,instance) (rate(kube_proxy_network_programming_duration_seconds_bucket[5m])))

# API Server rejected requests
sum(rate(apiserver_flowcontrol_rejected_requests_total[5m])) by (flow_schema)

# Business P99 latency (Istio)
histogram_quantile(0.99, sum by (le,destination_workload) (rate(istio_request_duration_milliseconds_bucket[1m])))