Graceful Shutdown in Kubernetes and Integration with AWS Load Balancer

In Q1 2020 SmartNews began promoting Kubernetes usage internally and encountered frequent complaints that during Deployment rolling upgrades the external AWS Load Balancer would lose traffic, prompting the need for graceful termination that first detaches traffic from Pods before deletion.

The article first discusses how to achieve graceful shutdown inside a Kubernetes cluster, referencing the official Kubernetes documentation and showing a diagram of the Pod deletion process.

When a Pod deletion request reaches the API server, two asynchronous workflows are triggered: one to close network connections and another to terminate the application. The network workflow removes the Pod from Endpoints, notifies the API server, and causes kube‑proxy and CoreDNS to update their rules. The application workflow has kubelet observe the event, execute any preStop hook, send a SIGTERM to the process, allow it to exit gracefully, and finally send SIGKILL if the process does not exit in time.

To guarantee graceful termination, the application workflow must run longer than the network workflow; common solutions include configuring a preStop hook that sleeps for a period after receiving SIGTERM so that connection cleanup can finish.

The article notes that if controller‑manager, CoreDNS, or kube‑proxy malfunction, traffic loss can still occur despite a long graceful shutdown period.

Section 02 addresses integration with AWS Load Balancer. Unlike the internal shutdown, external termination requires the AWS Load Balancer Controller to synchronize Ingress state with the AWS Load Balancer, which is subject to API latency and rate limiting, often causing traffic loss.

An architecture diagram of the AWS Load Balancer Controller is presented, followed by a list of its components:

Controller watches Ingress resources and syncs state to the AWS Load Balancer.

Ingress creates an ALB that can be configured via annotations.

Target Groups are created for each Service defined in the Ingress.

Listeners handle ports, also configurable via annotations.

Rules map paths to Services.

Two main solutions are proposed:

Use IP mode instead of instance mode for Ingress target type to reduce the number of AWS API calls caused by node additions or removals.

Deploy a sidecar container that, during the preStop hook, polls target group status and only exits when the target is draining, ensuring the correct order of traffic removal and application shutdown.

The article concludes that graceful shutdown is a universal requirement for Kubernetes platforms, emphasizing the declarative nature of the Kubernetes API and the need for applications to handle SIGTERM properly.

SmartNews mitigates AWS API rate‑limit issues by reducing unnecessary calls and converting asynchronous problems into synchronous ones to preserve execution order.

References: [1] Elastic Load Balancer: https://aws.amazon.com/cn/elasticloadbalancing/ [2] Graceful shutdown and zero‑downtime deployments in Kubernetes: https://learnk8s.io/graceful-shutdown [3] How AWS Load Balancer controller works: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/how-it-works/

For further discussion, readers are invited to comment on the SmartNews CN public account.