How to Gracefully Shut Down Kubernetes Pods Without Losing Traffic

When updating a deployment service, the old Pod is terminated and the new Pod takes over. If the old Pod has a long-running operation, we may want to kill it gracefully after the operation succeeds; otherwise traffic may be lost or errors increase during deployment.

Adding a graceful shutdown is simple; you can define a preStop hook and set an appropriate terminationGracePeriodSeconds. The article "K8S Pod流量的优雅无损切换实践" discusses best practices, but further improvements are needed.

Kubernetes kills a pod in five steps:

Pod transitions to Terminating state and stops receiving new traffic; containers keep running.

The preStop hook (a command or HTTP request) is executed inside the pod.

A SIGTERM signal is sent to the containers.

Kubernetes waits for the terminationGracePeriodSeconds (default 30 s) in parallel with the preStop hook and SIGTERM. If the period expires, it proceeds.

A SIGKILL signal is sent and the pod is removed.

In practice, define a preStop hook (often a 30 s sleep) to drain remaining traffic, then send SIGTERM so the service can close connections, deregister from service registries, etc.

Some wonder why traffic may still arrive after a pod is terminated and its endpoint removed. Because endpoint removal is asynchronous, which is why preStop runs before SIGTERM.

Graceful shutdown works if the service receives SIGTERM. Ideally a container runs a single process, but many real‑world containers run a shell script that starts a Java process plus side‑car processes, causing multiple processes.

The system sends SIGTERM to the main process and SIGKILL to remaining child processes. This is because most init scripts don’t forward signals, so the kernel ensures all processes are terminated.

You can trace system calls with strace -p pid to see signal handling.

If the main process isn’t the service itself, it may be force‑killed. The solution is to forward received signals to child processes so they all receive SIGTERM instead of SIGKILL.

Implementation example using a trap in a startup script:

#startup.sh
...
trap 'kill -TERM $child' TERM
nohup java $JAVA_OPTS -jar ./xxx.jar --server.port=8080 &
child=$!
wait $child
wait $child

Many frameworks already provide graceful shutdown, e.g., Spring’s CustomHealthCheck extending AbstractHealthIndicator, allowing you to override doHealthCheck(). You can also invoke such logic via a preStop HTTP call:

lifecycle:
  preStop:
    httpGet:
      path: /unhealthy
      port: http

After receiving the graceful shutdown signal, the service can perform cleanup tasks. Kubernetes itself is simple, but its low‑level behavior involves Linux kernel, processes, networking, storage, and other concepts that are essential to container technology.