From 502 Errors to Zero Loss: Production‑Ready Graceful Termination of Kubernetes Pods

Introduction: Why 502 Still Happens Even With Readiness

Many 502, 499, RST, and connection‑reset errors in production are not caused by buggy application code but by traffic removal and process termination occurring on different timelines.

Typical Failure Timeline

T0   Deployment creates new Pod
T1   Old Pod marked Terminating
T2   kubelet runs preStop
T3   EndpointSlice marks pod terminating / ready=false
T4   kube-proxy updates node forwarding table
T5   Ingress controller updates upstream
T6   External ALB/NLB removes backend
T7   kubelet sends SIGTERM to the container
T8   Application process exits

If any of the following occurs, failed requests appear: T8 < T6: process exits before the load balancer stops forwarding traffic. T7 continues to accept new requests without finishing them.

Long‑lived HTTP Keep‑Alive, HTTP/2 or gRPC connections are still reused.

Business logic such as message consumption, async tasks, or transaction commit is killed before completion.

Root Cause

Kubernetes only emits termination signals; it does not handle business‑level connection draining, long‑request finalization, message‑consumer shutdown, or external load‑balancer convergence.

Cross‑Layer Graceful Termination Mechanism

Control‑plane traffic removal.

Application layer rejects new traffic.

Connection layer drains in‑flight requests.

Business layer completes transactions and ensures idempotent cleanup.

Resource layer safely releases resources.

Production‑Level Goal

During pod shutdown, the pod should stop receiving new traffic, finish in‑flight requests, allow message consumption to pause, make remaining work interruptible, retryable, and observable.

Kubernetes Core Events

API Server writes deletionTimestamp.

Pod enters Terminating state.

EndpointSlice controller updates endpoint status.

kubelet executes preStop.

After preStop, kubelet sends SIGTERM.

Grace period terminationGracePeriodSeconds is waited.

If still running, SIGKILL is sent.

Key facts: preStop is blocking and consumes part of the grace period.

The application receives SIGTERM later than the pod deletion timestamp, so traffic may still be routed to the pod.

EndpointSlice States

terminating=true

ready=false

serving

may be kept depending on the scenario.

These states inform the service‑discovery layer but do not guarantee immediate data‑plane convergence for Ingress, cloud LB, NAT tables, or client connection pools.

Why Readiness Probe Is Not Enough

It only controls service‑discovery routing.

It does not drain long connections, stop new request acceptance, ensure in‑flight completion, pause message consumption, or halt async task creation.

Four‑Stage Termination Protocol

Declare Draining : set a draining flag, make readiness fail, return 503 Service Unavailable for new requests, pause message consumers, stop async task dispatch.

Traffic Drain : let control‑plane and traffic layers propagate (EndpointSlice update, Ingress upstream reload, cloud LB backend removal, client pool switch). Implemented via preStop calling an explicit drain endpoint.

In‑flight Completion : count inflight requests, close keep‑alive, send gRPC GOAWAY, wait for transactions, buffer flush, idempotent writes.

Force Close : after the deadline, cancel contexts, interrupt slow tasks, close connections, ensure the process finally exits (avoid staying in Terminating).

Grace Period Estimation

terminationGracePeriodSeconds >= traffic propagation time
                                 + max in‑flight request completion time
                                 + resource cleanup time
                                 + safety buffer

Practical formula (P99 values):

grace = LB removal P99 + request execution P99 + transaction/cache flush P99 + 20% buffer

Example values: LB removal = 12 s, request execution = 8 s, resource cleanup = 5 s, buffer = 5 s → terminationGracePeriodSeconds = 30 s, preStop wait = 12‑15 s, app shutdown timeout = 10‑12 s.

Production‑Ready Deployment Manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-gateway
  labels:
    app: payment-gateway
spec:
  replicas: 6
  minReadySeconds: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-gateway
  template:
    metadata:
      labels:
        app: payment-gateway
    spec:
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: payment-gateway
      containers:
        - name: app
          image: registry.example.com/payment-gateway:v3.4.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: APP_SHUTDOWN_TIMEOUT
              value: "20s"
            - name: APP_DRAIN_WAIT
              value: "12s"
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - >
                    wget -qO- --post-data='' http://127.0.0.1:8080/internal/drain || true;
                    sleep 12
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5
            failureThreshold: 24
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 2
            timeoutSeconds: 1
            failureThreshold: 1
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              cpu: "2"
              memory: "2Gi"
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-gateway-pdb
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: payment-gateway

Key configuration notes: maxUnavailable: 0 prevents capacity holes during rollout. minReadySeconds avoids immediate re‑selection of a newly ready pod.

Readiness probe period reduced to make traffic removal more responsive. preStop calls the local drain endpoint then sleeps for the estimated propagation time.

PDB protects against excessive simultaneous evictions.

Production‑Grade Go Implementation

package main

import (
    "context"
    "errors"
    "log"
    "net"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "sync/atomic"
    "syscall"
    "time"
)

type DrainManager struct {
    draining   atomic.Bool
    inflight   atomic.Int64
    activeConns atomic.Int64
    wg        sync.WaitGroup
}

func (d *DrainManager) IsDraining() bool { return d.draining.Load() }

func (d *DrainManager) StartDrain() {
    if d.draining.CompareAndSwap(false, true) {
        log.Println("drain mode enabled")
    }
}

func (d *DrainManager) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if d.IsDraining() && r.URL.Path != "/healthz" && r.URL.Path != "/readyz" && r.URL.Path != "/internal/drain" {
            http.Error(w, "server is draining", http.StatusServiceUnavailable)
            return
        }
        d.inflight.Add(1)
        d.wg.Add(1)
        defer func() {
            d.inflight.Add(-1)
            d.wg.Done()
        }()
        next.ServeHTTP(w, r)
    })
}

func (d *DrainManager) ConnState(c net.Conn, state http.ConnState) {
    switch state {
    case http.StateNew:
        d.activeConns.Add(1)
    case http.StateClosed, http.StateHijacked:
        d.activeConns.Add(-1)
    }
}

func main() {
    drain := &DrainManager{}
    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK); _, _ = w.Write([]byte("ok")) })
    mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
        if drain.IsDraining() { http.Error(w, "not ready", http.StatusServiceUnavailable); return }
        w.WriteHeader(http.StatusOK); _, _ = w.Write([]byte("ready"))
    })
    mux.HandleFunc("/internal/drain", func(w http.ResponseWriter, r *http.Request) { drain.StartDrain(); w.WriteHeader(http.StatusOK); _, _ = w.Write([]byte("draining")) })
    mux.HandleFunc("/api/pay", func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        select {
        case <-time.After(1500 * time.Millisecond):
            _, _ = w.Write([]byte("payment accepted"))
        case <-ctx.Done():
            http.Error(w, "request canceled", http.StatusRequestTimeout)
        }
    })
    server := &http.Server{Addr: ":8080", Handler: drain.Middleware(mux), ReadHeaderTimeout: 3 * time.Second, ReadTimeout: 5 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 30 * time.Second, ConnState: drain.ConnState}
    go func() { log.Println("http server started on :8080"); if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { log.Fatalf("listen error: %v", err) } }()
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    <-sigCh
    log.Println("shutdown signal received")
    drain.StartDrain()
    server.SetKeepAlivesEnabled(false)
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()
    go func() {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-shutdownCtx.Done():
                return
            case <-ticker.C:
                log.Printf("draining: inflight=%d activeConns=%d", drain.inflight.Load(), drain.activeConns.Load())
            }
        }
    }()
    if err := server.Shutdown(shutdownCtx); err != nil { log.Printf("graceful shutdown timeout: %v", err) }
    waitCh := make(chan struct{})
    go func() { defer close(waitCh); drain.wg.Wait() }()
    select {
    case <-waitCh:
        log.Println("all inflight requests completed")
    case <-shutdownCtx.Done():
        log.Println("shutdown deadline reached, force exit")
    }
    if err := closeResources(); err != nil { log.Printf("close resources error: %v", err) }
    log.Println("server exited gracefully")
}

func closeResources() error {
    // close DB pools, flush logs, stop consumers, report final metrics, etc.
    time.Sleep(500 * time.Millisecond)
    return nil
}

This implementation differs from a naive example because it:

Rejects new requests at the business layer, not only via readiness.

Explicitly tracks inflight requests instead of relying solely on server.Shutdown.

Disables keep‑alive to prevent backend reuse.

Enforces a deadline and forces termination if the grace period expires.

Spring Boot Production Implementation

Simply enabling server.shutdown=graceful is insufficient. Production code must also:

Switch readiness to fail.

Reject new traffic.

Pause message consumers.

Stop async task dispatch.

Ensure transaction commit and cache flush before exit.

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 20s
management:
  endpoint:
    health:
      probes:
        enabled: true

Handling gRPC, Message Consumers, and Async Tasks

For gRPC services, call GracefulStop() after setting readiness to fail, wait for connections to close, then invoke Stop() on timeout.

For Kafka (or RocketMQ/RabbitMQ) consumers, the safe shutdown order is:

Set drain state.

Call pause(partitions).

Wait for current batch/in‑flight handlers to finish.

Commit offsets.

Close the consumer.

Incorrectly closing the consumer first leads to uncommitted offsets, duplicate consumption, and downstream alerts.

Async thread pools and scheduled executors must stop dispatching new work before the process exits.

Ingress, Service Mesh, and Cloud Load Balancer Differences

Ingress controllers (NGINX, Kong, APISIX) need to watch for upstream reload latency and keep‑alive reuse. Cloud LB health‑check cycles and backend removal delays are common sources of intermittent 502s. Service‑mesh sidecars (Istio/Envoy) must drain listeners after the application enters drain, otherwise the sidecar may cut the network before the app finishes.

Observability Metrics

app_draining_state
app_inflight_requests
app_active_connections
app_shutdown_duration_seconds
app_shutdown_rejected_requests_total
app_shutdown_forced_total
consumer_inflight_messages
consumer_pause_total

Key dashboards should monitor 5xx ratios during rollout, inflight request decay curves, shutdown duration percentiles, forced shutdown counts, and message backlog/offset latency.

Real‑World Payment Service Scenario

A typical payment gateway handles three traffic types: synchronous HTTP orders, asynchronous callbacks, and Kafka‑based result consumption. The zero‑loss shutdown sequence is:

Fail /readyz, call /internal/drain, pause Kafka consumer, stop scheduled tasks.

Wait for EndpointSlice, Ingress, and cloud LB convergence.

Allow in‑flight HTTP requests, message handling, and transaction commits to finish.

Shutdown thread pools, connection pools, and finally exit the process.

If the deadline expires, the system must rely on idempotent keys, compensation jobs, client retry strategies, and audit logs to guarantee no business loss.

Common Pitfalls and Anti‑Patterns

Only setting preStop sleep without a coordinated drain endpoint.

Exiting immediately on SIGTERM before traffic is fully removed.

Embedding heavy external dependencies (DB, Redis, downstream services) in readiness probes, causing mass evictions on transient failures.

Using a grace period without real‑world load‑test measurements.

Ignoring sidecar lifecycles (log agents, mesh proxies) that can cut network connections early.

Validation Practices

Four validation types are recommended:

Rolling‑update load test (10‑20 min) – watch 5xx spikes, p99 latency, and stray requests to terminating pods.

Long‑request test – issue 10‑30 s requests and verify graceful completion or proper cancellation.

Consumer test – trigger pod drain while processing messages and verify offset commits and idempotency.

Node‑drain / spot‑instance eviction drills – ensure the same graceful flow works for infrastructure‑initiated evictions.

Recommended Baselines

Kubernetes Baseline

maxUnavailable: 0

minReadySeconds: 5‑15

terminationGracePeriodSeconds: 30‑60

readinessProbe.periodSeconds: 2‑3

Configure a PodDisruptionBudget.

Application Baseline

Expose /readyz, /healthz, and /internal/drain endpoints.

Maintain a draining flag.

Reject new requests when draining.

Track inflight request count.

Pause message consumption and async task dispatch.

Set a shutdown deadline and enforce forced exit.

Observability Baseline

5xx rollout dashboard.

Inflight request drain curve.

Forced shutdown alerts.

Terminating‑pod request sampling logs.

Architecture Baseline

Idempotent writes.

Retry‑capable clients.

Transaction and compensation mechanisms.

Long‑connection convergence strategies.

Conclusion

Kubernetes only provides the pod‑lifecycle hooks; achieving near‑zero 502s requires a full chain: declare drain, remove traffic, stop accepting new work, finish in‑flight work, then exit. Only when control‑plane, application, and infrastructure layers are coordinated can you claim production‑grade graceful termination.