Mastering Graceful Shutdown in Kubernetes with Spring Boot and Nacos

Concept of Graceful Shutdown

Graceful (or lossless) shutdown is a controlled termination process that ensures data safety, prevents errors, and keeps the system stable before a service stops.

Backup data : Flush in‑memory state or caches to persistent storage.

Stop accepting new requests .

Process in‑flight requests and finish ongoing work.

Notify dependent components (e.g., service discovery, downstream caches).

Wait for all elements to exit safely before terminating the process.

In microservice environments additional actions such as stopping scheduled tasks and message‑queue consumption are required.

Kubernetes Termination Flow

Network rule enforcement

Kube‑apiserver receives the kubectl delete pod request and marks the pod as Terminating in etcd.

The endpoint controller removes the pod IP from the corresponding Endpoints object.

Kube‑proxy updates iptables so that traffic no longer routes to the pod.

Container removal

Kubelet cleans up container resources (storage, network).

A PreStop hook can be defined to wait for traffic to drain.

Kubelet sends SIGTERM to the container.

If the container does not exit within the default terminationGracePeriodSeconds (30 s), Kubelet sends SIGKILL to force termination.

Spring Boot + Nacos Example

A pod runs a Spring Boot application registered with Nacos. The PreStop hook performs two actions:

Deregister the service from Nacos.

Sleep for 35 seconds to give downstream caches time to refresh.

The pod’s terminationGracePeriodSeconds is set to 35 seconds.

Problems Observed

Spring Boot shutdown takes only ~2 seconds, which is insufficient for completing asynchronous tasks, MQ consumption, or scheduled jobs.

The 35 s sleep plus the deregistration time exceeds the configured grace period; Kubelet adds a one‑time extra 2 seconds and then kills the pod with SIGKILL.

Nacos service‑change propagation is not truly real‑time: HTTP polling can take up to 10 seconds, and Ribbon’s default cache refresh interval is 30 seconds.

Optimization Suggestions

Reduce the post‑deregistration sleep. The safe window should be service‑change time + Ribbon cache refresh (≈40 seconds) in the worst case.

Set terminationGracePeriodSeconds to a value slightly larger than PreStop duration plus Spring Boot shutdown time. A practical setting is 10 s + 30 s (Spring Boot’s default graceful‑shutdown buffer).

Sample Listener to Refresh Ribbon Cache on Nacos Instance Change

@Component
@Slf4j
public class NacosInstancesChangeEventListener extends Subscriber<InstancesChangeEvent> {
    @Resource
    private SpringClientFactory springClientFactory;

    @PostConstruct
    public void registerToNotifyCenter() {
        NotifyCenter.registerSubscriber(this);
    }

    @Override
    public void onEvent(InstancesChangeEvent event) {
        String service = event.getServiceName();
        // service format: DEFAULT_GROUP@@demo  -> ribbonService: demo
        String ribbonService = service.substring(service.indexOf("@@") + 2);
        log.info("#### Received Nacos instance change event:{} ribbonService:{}", event.getServiceName(), ribbonService);
        ILoadBalancer loadBalancer = springClientFactory.getLoadBalancer(ribbonService);
        if (loadBalancer != null) {
            ((ZoneAwareLoadBalancer<?>) loadBalancer).updateListOfServers();
            log.info("Refreshed Ribbon cache for service:{}", ribbonService);
        }
    }

    @Override
    public Class<? extends com.alibaba.nacos.common.notify.Event> subscribeType() {
        return InstancesChangeEvent.class;
    }

    @Override
    public boolean scopeMatches(InstancesChangeEvent event) {
        return true;
    }
}

Further Optimizations

MQ and Scheduled Tasks

The shutting‑down service can listen to its own Nacos deregistration event. When it detects that it is being deregistered, it should stop consuming MQ messages and suspend scheduled jobs, ensuring a cleaner shutdown than handling these tasks only in a generic graceful‑shutdown hook.

Traffic Control

If the cluster does not use Kubernetes‑level pod traffic control, a Spring Cloud Gateway can act as the edge proxy. The gateway should also listen to Nacos deregistration events to refresh its Ribbon cache and stop routing traffic to the terminating service.

Key Take‑aways

Configure terminationGracePeriodSeconds to exceed the longest expected shutdown activity (service‑discovery propagation, cache refresh, in‑flight tasks).

Implement custom shutdown hooks in Spring Boot to persist state, finish in‑flight work, and stop background consumers (MQ, scheduled tasks).

Make APIs idempotent to tolerate retries during the shutdown window.

When using Spring Boot’s actuator /shutdown, configure thread‑pool executors to wait for tasks to complete:

threadPoolTaskExecutor.setWaitForTasksToCompleteOnShutdown(true);
threadPoolTaskExecutor.setAwaitTerminationSeconds(30);

Following these practices allows services to terminate without data loss, without abrupt request failures, and with minimal impact on the overall system.