Spring Boot Graceful Shutdown: SIGTERM to Container Exit, Request Draining & Kubernetes Timing

From SIGTERM to Container Shutdown: The Call Chain

Both docker stop and kubectl delete pod send SIGTERM. The JVM does not exit immediately; it starts a dedicated thread to run all registered shutdown hooks, and only terminates after they complete. kill -9 sends SIGKILL, which the kernel handles directly without executing any hooks. Multiple JVM-level hooks run concurrently with no guaranteed order; Spring's internal Lifecycle phases provide ordering, but these are separate concerns.

Spring Boot registers a hook named SpringContextShutdownHook that simply calls context.close(). This method splits into two halves: stop() stops all Lifecycle beans by phase (web server draining happens here), and destroy() destroys singletons, runs destroy callbacks, and closes data sources and thread pools. The gap between these two phases is where CannotGetJdbcConnectionException originates — the web server has stopped accepting requests, but business threads are still running while the data source waits in the destruction queue.

// SpringApplicationShutdownHook equivalent logic, unified from 2.4
Runtime.getRuntime().addShutdownHook(
    new Thread(() -> {
        // SIGKILL never reaches this line; process disappears with no cleanup chance
        context.close();
    }, "SpringContextShutdownHook"));

Graceful Switch and Phase Timeouts

server.shutdown

was added in Spring Boot 2.3 with default immediate; since 3.4 the default is graceful. The release notes state: embedded web server graceful shutdown is now enabled by default; to restore old behavior set server.shutdown=immediate. Projects upgrading across 3.3 to 3.4 will see shutdown behavior change without any config modification.

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

The timeout defaults to 30 seconds per phase, not for the whole shutdown. DefaultLifecycleProcessor.setTimeoutPerShutdownPhase javadoc: "maximum time allocated for the shutdown of any single phase; a phase is a group of SmartLifecycle beans with the same phase value." If an application has three phases, theoretical max time can exceed 90 seconds, yet most people only consider single-phase duration.

Kubernetes terminationGracePeriodSeconds also defaults to 30 seconds, colliding with Spring's default. Two 30-second budgets stacked means SIGKILL may arrive near the end of draining, wasting the earlier wait. Undertow behaves differently: it keeps accepting connections but immediately returns 503, whereas Tomcat, Jetty, and Reactor Netty stop accepting at the network layer (connection refused/reset). Choice depends on whether upstream retries on status codes.

Connector Pause and In-Flight Requests

Tomcat's graceful shutdown has three steps: pause the Connector (Acceptor thread stops pulling new connections), poll worker threads to count active requests, wait for count to reach zero (or timeout), then stop the connector and thread pool. The pause happens at the network layer, not the application layer, so new requests during draining never reach DispatcherServlet — no Spring interceptor, filter, or @ControllerAdvice can see them. Returning a friendly 503 during drain must be handled by an upstream gateway or the server's connection rejection.

Request counting relies on worker thread activity. A long-polling or WebSocket connection keeps the count non-zero, forcing drain to wait until timeout, then abort with a batch of interruption exceptions sharing the same timestamp. Such long-connection workloads are unsuitable for graceful mode; they should use a separate port or be actively disconnected in a preStop hook.

Thread Pool and Business Component Shutdown Order

Web request draining may finish while business thread pools are still running. ThreadPoolTaskExecutor has two unfriendly defaults: waitForTasksToCompleteOnShutdown=false and awaitTerminationSeconds=0. With both unset, shutdown() calls shutdownNow(), interrupting running tasks and discarding queued tasks silently — a reconciliation message or settlement notification disappears without a log line.

@Bean("settleExecutor")
public ThreadPoolTaskExecutor settleExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(4);
    executor.setMaxPoolSize(8);
    executor.setQueueCapacity(200);
    executor.setThreadNamePrefix("settle-");
    // Default false triggers shutdownNow, discarding queued tasks
    executor.setWaitForTasksToCompleteOnShutdown(true);
    executor.setAwaitTerminationSeconds(20);
    return executor;
}

awaitTerminationSeconds

must be set; default 0 means no wait. If awaitTermination returns false, Spring calls shutdownNow() again, dropping remaining tasks and logging a warning with the count of unexecuted tasks. This value must exceed the longest task duration with margin; setting 5 seconds for a 15-second task wastes 5 seconds before force-kill.

2026-10-11 09:32:38.517 WARN  [SpringContextShutdownHook] o.s.s.concurrent.ExecutorConfigurationSupport : Executor did not terminate in the specified time.
2026-10-11 09:32:38.518 WARN  [SpringContextShutdownHook] o.s.s.concurrent.ExecutorConfigurationSupport : Executor was abruptly shut down. 7 tasks will not be executed.

After thread pools, message consumers, scheduled tasks, and local cache refreshers are often forgotten. Their stop order is determined by SmartLifecycle phase. SmartLifecycle.DEFAULT_PHASE is Integer.MAX_VALUE; higher phase stops first. In Spring Boot 2.5, WebServerGracefulShutdownLifecycle used DEFAULT_PHASE (max), causing web server to drain before consumers stopped — order reversed. Spring Boot 3.0 split these into separate phases, leaving room for business components.

@Component
public class SettlementConsumer implements SmartLifecycle {
    // Phase earlier than web server drain, stop pulling first
    private static final int PHASE = SmartLifecycle.DEFAULT_PHASE - 512;
    private final MessageListenerContainer container;
    private volatile boolean running;
    public SettlementConsumer(MessageListenerContainer container) { this.container = container; }
    @Override public void start() { container.start(); this.running = true; }
    @Override public void stop() { container.stop(); this.running = false; }
    @Override public boolean isRunning() { return this.running; }
    @Override public int getPhase() { return PHASE; }
}

Using @PreDestroy to stop consumers runs during bean destruction, after web server shutdown and possibly after data source closure, leading to high failure rates when calling downstream services.

Three Time Constants in Kubernetes

Traffic draining and signal sending are distinct. After a Pod is marked terminating, Endpoints removal waits for kube-proxy and iptables updates (hundreds of milliseconds to seconds). During this window, requests still arrive while the container may already be pausing its connector, resulting in connection resets with no server-side logs.

Three time constants must form a non-conflicting chain; if any is smaller than the others, SIGKILL lands mid-process.

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: order-service
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]
    readinessProbe:
      httpGet:
        path: /actuator/health/readiness
        port: 8080
      periodSeconds: 5

preStop

sleep buffers traffic draining, waiting for routing rules to propagate before the process receives SIGTERM. terminationGracePeriodSeconds is the total budget from SIGTERM to SIGKILL; it must exceed preStop plus the sum of per-phase timeouts. The example uses 10s preStop + 30s phase timeout, leaving 20s margin for data source closure and JVM exit.

During preStop the process hasn't received SIGTERM yet, so business runs normally. If routing hasn't converged, requests continue to arrive and are caught by the subsequent drain window, appearing as a mix of successes and connection refusals — the familiar spike in monitoring.

A counterexample: keep terminationGracePeriodSeconds at default 30, timeout-per-shutdown-phase at default 30, and preStop sleep 10 seconds. Budget is 30, actual need is 40; SIGKILL hits halfway through draining, wasting all prior wait time.

Reproducible Drain Verification

Minimal setup: a slow endpoint, a thread pool task, and a bash script that continuously sends requests while capturing HTTP codes.

@RestController
public class OrderController {
    private final OrderService orderService;
    @GetMapping("/order/{id}")
    public String query(@PathVariable String id) throws InterruptedException {
        log.info("开始查询订单 {}", id);
        // Simulate 3-second downstream response to observe drain effectiveness
        Thread.sleep(3000);
        orderService.notifyAsync(id);
        log.info("查询完成 {}", id);
        return "ok";
    }
}

#!/usr/bin/env bash
PID=$(jps -l | grep drain-demo | awk '{print $1}')
for i in $(seq 1 300); do
  curl -s -o /dev/null -w "%{http_code}
" --max-time 10 "http://127.0.0.1:8080/order/${i}" >> codes.txt &
  sleep 0.02
done
sleep 2
kill -15 "$PID"
wait
sort codes.txt | uniq -c

Run once with graceful shutdown off, once with it on; the codes.txt difference is clear.

# server.shutdown=immediate
    187 200
     19 000   # curl couldn't connect or connection reset
     94 500   # server executed halfway then interrupted

# server.shutdown=graceful, timeout 30s
    298 200
      2 000

000

is curl's own code meaning no response received. Immediate mode splits requests into two failure types: connection refused and mid-execution interruption. Graceful mode completes nearly all requests; the two 000s are requests rejected by the connector pause.

Thread pool behavior must be verified separately. Remove waitForTasksToCompleteOnShutdown, repeat the experiment, and observe the log line tasks will not be executed. That number equals the notifications lost during a deploy — small enough to go unnoticed until reconciliation reveals missing settlement records.