Spring Boot Graceful Shutdown & Health Checks: Zero-Downtime Deployment Deep Dive

1. Problem Definition: Three Types of Request Loss During Deployments

The article opens with a common scenario: during a rolling update, 502/504 errors spike in the deployment window but application logs show no exceptions, no errors, and no OOM. These alerts are often dismissed as "normal deployment jitter." However, the user experience is failed requests (e.g., order submission spins then white screen) that succeed on retry. The author identifies three distinct causes of request loss, each requiring a different solution:

Type 1: Traffic still arriving, process already gone. Kubernetes sends SIGTERM, Spring context closes immediately, Tomcat acceptor stops accepting, in-flight requests are cut (ECONNRESET). Worse, kube-proxy/Ingress may not have removed the Pod from endpoints yet, so upstream keeps sending traffic to a dying instance → 502.

Type 2: Process alive, but dependencies already unavailable. Spring container closes "outside-in": Web layer first, then business beans, then data sources/connection pools. If a custom @PreDestroy closes an MQ consumer out of order, the consumer thread may still pull messages while its DataSource is already closed → "Connection is closed" / "HikariPool has been shut down" errors, transaction rollbacks, message requeue, potential data inconsistency.

Type 3: Health check lies. /actuator/health returns UP, but the app cannot actually handle requests: thread pool exhausted, DB connections unavailable, downstream timeouts. Teams often treat health as "process alive" probe, but Kubernetes uses it as "ready for traffic" probe. Misaligned semantics turn rolling updates into full outages.

A comparison table summarizes the four failure modes (hard cutoff, routing lag, dependency-first, probe lying) with their symptoms and root causes.

2. Technical Analysis: Five Components and Their Boundaries

Spring Boot doesn't invent a new lifecycle protocol; it composes existing abstractions: Spring lifecycle, JDK shutdown hook, Servlet container stop semantics, exposed via a single property. Understanding the five components is more valuable than memorizing config keys:

SpringApplication – registers JVM shutdown hook.

ShutdownHook – receives signal → closes context.

SmartLifecycle – orchestrates bean stop order (higher phase stops first). Web container typically has max phase → stops first (stops traffic before tearing down internals).

WebServer – stops accepting + drains requests.

HealthEndpoint – emits health state for probes.

A critical detail: JDK does not guarantee order among multiple shutdown hooks; they run concurrently. If you register your own hook to close a connection pool, it may race with Spring's hook.

3. Core Flow: The 30 Seconds After SIGTERM

3.1 Phase 1: Hook Awakened (0–tens of ms)

JVM starts all registered shutdown hook threads. Spring Boot registers a SpringApplicationShutdownHook at the end of SpringApplication.run(). It iterates all ApplicationContext instances, closes each in a separate thread with a timeout (default 30s). If a context close times out, the thread is not interrupted ; the hook simply stops waiting. The thread's stack is lost — debugging requires jstack at the exact shutdown moment.

3.2 Phase 2: Container Close Five Steps (tens of ms – seconds)

AbstractApplicationContext.doClose()

executes a fixed sequence that is a design contract:

Publish

ContextClosedEvent</strong> – lets external components (registry, config center, metrics) know "I'm going down" first.</li>
<li><code>lifecycleProcessor.onClose()

→ SmartLifecycle.stop(Runnable cb) – stops beans by phase descending. This is the only ordered, interruptible phase you can control. @PreDestroy runs in step 3, ordered by dependency graph, not controllable. destroyBeans() – @PreDestroy / DisposableBean.destroy, dependencies destroyed before dependents.

Reset common caches.

Close BeanFactory – after this, any async thread accessing container gets IllegalStateException.

3.3 Phase 3: Web Container Drain (the real time consumer)

Tomcat's graceful shutdown is not a sleep but: connector.pause() (stop accepting new connections) + poll active request count every 1 second (hardcoded in 2.x) until zero or timeout. Three subtle points: connector.pause() only stops accept; established keep-alive connections continue to be read — occasional requests after pause are normal.

Polling counts active requests , not threads. Async Servlet ( DeferredResult, WebFlux) uses a separate counter; misconfigured container can wait until timeout despite zero apparent requests.

On timeout: force stop , interrupting running threads. Log shows "Graceful shutdown failed, forcing" — indicates grace period too short.

3.4 Phase 4: Timeout & SIGKILL

If Kubernetes terminationGracePeriodSeconds < Spring's timeout-per-shutdown-phase, SIGKILL arrives before Spring finishes draining — graceful shutdown config becomes completely ineffective.

3.5 Time Accounting: Three Timeouts Must Nest

preStop sleep 8s
+ Spring grace 30s
+ container stop margin 5s
──────────────────────────
= 43s MUST BE < terminationGracePeriod
Default terminationGracePeriodSeconds = 30
→ MUST explicitly increase, otherwise guaranteed SIGKILL

Most common accident: server.shutdown=graceful + 30s grace + preStop sleep 8s but terminationGracePeriodSeconds left at default 30s → only 22s actual drain time; slightly slow long requests get SIGKILLed.

4. Core Source Code: Six Classes That Decide Success/Failure

4.1 SpringApplicationShutdownHook

Registers hook, snapshots contexts, closes each in a thread with join(timeout) then interrupt(). The 30s context-close timeout is hardcoded in Spring, not configurable — even if you set phase timeout to 60s, it won't run past 30s.

4.2 DefaultLifecycleProcessor

Groups beans by phase (descending), stops each group sequentially; beans within same phase stop concurrently. This gives a controllable orchestration lever: to make A stop completely before B, assign different phases.

Reference phase values: Integer.MAX_VALUE WebServer 2147483647-1 (suggested) custom "stop after Web" 0 default 0 ~ MAX/2 business components Integer.MIN_VALUE stop earliest

4.3 WebServerGracefulShutdownLifecycle (3.x bridge bean)

New in 3.x, makes graceful shutdown itself a SmartLifecycle with phase DEFAULT_PHASE - 1024 (slightly before WebServer). In 2.x, graceful shutdown was inside WebServerStartStopLifecycle.stop(), causing conflict between onClose order and phase ordering. Extraction enables precise phase sorting and custom WebServer override.

4.4 SmartLifecycle: Two Implementation Essentials

@Component
public class MqConsumerLifecycle implements SmartLifecycle {
    private volatile boolean running = true;
    public int getPhase() { return 0; } // 0: stops AFTER Web (Web = MAX_VALUE)
    public void stop(Runnable callback) {
        try {
            consumer.pause();           // 1. stop pulling new messages
            consumer.awaitDrain(Duration.ofSeconds(10)); // 2. wait in-flight
        } finally {
            callback.run(); // MUST call!
        }
    }
    public boolean isRunning() { return running; }
}

Essential 1: callback.run() must be called, and in finally. Omitting it makes the whole phase group wait until timeout, wasting grace period.

Essential 2: Only override stop(Runnable); overriding no-arg stop() forces synchronous path, losing timeout protection.

4.5 HealthEndpoint: Three-State Model

Status enum: UP, DOWN, OUT_OF_SERVICE, UNKNOWN. Many teams miss OUT_OF_SERVICE — semantics: "I know I'm unhealthy, but don't restart me, just don't send traffic." This is exactly what active drain needs. Kubernetes readiness treats non-UP as failure; liveness only treats DOWN / non-2xx as failure. From 3.x, liveness and readiness use separate probe groups via management.endpoint.health.group.liveness.include; default liveness includes only livenessState and ping — deliberately excludes DB to prevent "DB blip → whole cluster restart → restart storm → DB pressure ↑ → avalanche."

4.6 AvailabilityChangeEvent: Active Drain Hook

@RestController
public class DrainController {
    private final ApplicationEventPublisher publisher;
    @PostMapping("/internal/drain")
    public void drain() {
        publisher.publishEvent(new AvailabilityChangeEvent(this, ReadinessState.REFUSING_TRAFFIC));
        new Thread(() -> { sleep(10_000); System.exit(0); }).start();
    }
}

Practical "active drain" pattern: flip readiness first, wait, then exit . Turns "guess routing convergence time" into "confirm state", more reliable than pure preStop sleep. Cost: requires internal call orchestrated by deployment platform.

5. Health Checks: From Ping to Semantic Probes

5.1 Three States & Three Probes Mapping

liveness : "Are you alive?" Failure → container restart.

readiness : "Can you take traffic?" Failure → endpoint removed.

startup : "Started yet?" Failure → keep waiting.

Key: Only liveness failure triggers restart. Any dependency that may transiently blip (DB, Redis, downstream HTTP) must never be in liveness — otherwise a DB failover restarts the whole cluster, amplifying local fault to global outage.

5.2 HealthIndicator Implementation Rules

@Component("orderDownstream")
public class DownstreamHealthIndicator implements HealthIndicator {
    private final RestClient client;
    public Health health() {
        try {
            Status s = client.get().uri("/ping").retrieve().body(Status.class);
            return Health.up().withDetail("status", s).build();
        } catch (Exception e) {
            return Health.down().withDetail("error", e.getClass().getSimpleName()).build();
        }
    }
}

Three hard requirements: timeout short (≤1s), no side effects (no big table scans), failure must include detail (otherwise debugging sees only DOWN). HealthIndicators are called at probe frequency; any heavy logic becomes a pressure source.

5.3 Probe Groups: Different Probes Check Different Things

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState,ping
        readiness:
          include: readinessState,db,orderDownstream
      show-details: when-authorized

management.endpoint.health.probes.enabled=true

auto-enables in K8s (detects *_SERVICE_HOST env vars), exposing /actuator/health/liveness and /actuator/health/readiness.

5.4 Correct Custom Health State

Don't return null or throw; explicitly return Health.up(), Health.down(), or Health.outOfService(). Health.status() takes worst state in group: one DOWN → whole group DOWN; OUT_OF_SERVICE priority between DOWN and UP. Understanding this ordering enables "dependency down → drop traffic but no restart" combos.

6. Kubernetes Coordination: Probe Parameter Tuning

6.1 Ready-to-Use Probe Config

startupProbe:
  httpGet: {path: /actuator/health/liveness, port: 8080}
  failureThreshold: 30
  periodSeconds: 5  # max wait 150s
livenessProbe:
  httpGet: {path: /actuator/health/liveness, port: 8080}
  periodSeconds: 10
  failureThreshold: 3  # 30s to declare dead
readinessProbe:
  httpGet: {path: /actuator/health/readiness, port: 8080}
  periodSeconds: 5
  failureThreshold: 2  # 10s to drop traffic

Readiness more sensitive than liveness because dropping traffic is reversible/low-cost; restart is irreversible/high-cost . Asymmetric fault tolerance: prefer fast traffic drop over premature restart.

6.2 Parameter Arithmetic

Must satisfy:
gracePeriod >
  preStop sleep
  + Spring timeout-per-shutdown-phase
  + readiness detection latency (~2×period)
  + 5s margin
Example: 8 + 30 + 10 + 5 = 53s
→ terminationGracePeriodSeconds: 60

Don't just set gracePeriod to 300s — slows rollout (each Pod waits full). Correct: measure P99 request latency, set grace to 2–3× that.

6.3 When to Add preStop Sleep

Needed: Ingress/gateway has route cache (Nginx, Envoy, cloud SLB) — removal not instant.

Needed: Service + kube-proxy iptables mode (sync has period).

Optional: Active drain endpoint used (flip readiness → wait).

Optional: IPVS mode + client-side retry + connection pool health checks.

Even when optional, keep a 3–5s short preStop — cheap insurance, costs only a few seconds per deploy.

7. Complete Practice: Order Service Refactor

Order service depends on MySQL, Redis, Kafka consumer; runs on K8s behind Nginx Ingress. Pre-refactor: ~0.3% 5xx per deploy. Post-refactor: 0.

7.1 Config (application.yml)

server:
  shutdown: graceful
  tomcat:
    keep-alive-timeout: 15s
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState,ping
        readiness:
          include: readinessState,db,redis
  endpoints:
    web:
      exposure:
        include: health,info,prometheus

7.2 Destruction Order Orchestration

@Component
public class KafkaDrainLifecycle implements SmartLifecycle {
    private volatile boolean running = true;
    public int getPhase() { return 0; } // smaller than Web → Web stops first
    public void stop(Runnable cb) {
        try {
            listenerContainer.pause();
            listenerContainer.stop(Duration.ofSeconds(15));
            running = false;
        } finally { cb.run(); }
    }
    public boolean isRunning() { return running; }
}

Why "stop Web first, then consumer"? Consumer logic may depend on Web-layer local caches or Feign clients; reverse would cause incoming orders to write DB but fail to emit messages — true data inconsistency.

7.3 Startup Readiness Semantics

@Component
public class WarmupRunner implements ApplicationRunner {
    private final ApplicationEventPublisher pub;
    public void run(ApplicationArguments a) {
        warmup(); // prewarm pools, caches, routes
        pub.publishEvent(new AvailabilityChangeEvent(this, ReadinessState.ACCEPTING_TRAFFIC));
    }
}

Boot auto-sets readiness to ACCEPTING at ApplicationReadyEvent, so this isn't mandatory. But if you have cold-start warmup (e.g., load rules to local cache), must gate traffic until warmup completes , else first requests timeout.

7.4 Deployment Side (Deployment.yaml)

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: order
    lifecycle:
      preStop:
        exec:
          command: ["sh","-c","sleep 8"]
    startupProbe: ...
    livenessProbe: ...
    readinessProbe: ...

7.5 Before/After Comparison

Metric          Before    After
────────────────────────────────────
Deploy 5xx      0.3%      0
ECONNRESET      Yes       No
Duplicate consume Occasional None
Deploy duration ~3 min    ~4 min
Data inconsistency complaints 1-2/month 0

Deploy time increased 1 minute for zero loss — a worthwhile trade in any system with real transactions.

8. Eight Pitfalls & Troubleshooting Handbook

8.1 Eight High-Frequency Pitfalls

Only graceful, no gracePeriod tune. SIGKILL beats drain → config useless.

Liveness checks DB. DB blip → full cluster restart → avalanche.

Custom hook races Spring hook. Connection pool closed early → "HikariPool shut down".

SmartLifecycle forgets callback.run(). Whole phase group waits timeout, grace period wasted.

@PreDestroy does heavy work. Not phase-protected; interrupted on timeout.

HealthIndicator no timeout. Probe requests pile up, saturate thread pool.

Readiness always UP. Traffic hits closing instance → 502 spike.

System.exit() for drain. Bypasses all hooks = hard kill.

8.2 Troubleshooting: Four Symptoms & Actions

Symptom: Deploy 502
Check: endpoint removal timing vs SIGTERM timing
Fix: add preStop sleep / active drain

Symptom: Graceful shutdown aborted
Check: timeout-per-shutdown-phase
Fix: increase grace + investigate slow requests

Symptom: HikariPool has been shut down
Check: custom shutdown hook present?
Fix: remove, use SmartLifecycle

Symptom: Readiness stuck red
Check: /actuator/health/readiness details
Fix: show-details=always to pinpoint component

8.3 Two Debug Switches

# 1. Full health group details
management.endpoint.health.show-details=always

# 2. Lifecycle event timing
logging.level.org.springframework.context.support.DefaultLifecycleProcessor=DEBUG

The second is especially useful: prints each phase's start/stop duration — instantly reveals what's slowing shutdown. Many "shutdown slow" root causes hide in those millisecond numbers.

9. Learning Path & 12 Interview Questions

9.1 Four-Stage Path

Stage 1 | Can use (1 day): graceful switch + probe groups working. Acceptance: kill -15 → no 5xx.

Stage 2 | Can tune (1 week): preStop + gracePeriod time accounting; asymmetric three-probe design. Acceptance: rolling deploy zero 5xx.

Stage 3 | Can orchestrate (2 weeks): SmartLifecycle phase design; consumers / scheduled tasks / cache warmup. Acceptance: destruction order verifiable.

Stage 4 | Can design (ongoing): active drain endpoint + deployment platform orchestration; probe & alerting integration. Acceptance: fault injection causes zero damage.

9.2 Twelve Interview Questions (with answers)

Essence of graceful shutdown? Pause accept + poll in-flight request count to zero, not sleep.

Why only server.shutdown=graceful not enough? Routing convergence lag; need preStop or active drain.

Liveness vs readiness? Former fails → restart; latter fails → drop traffic; fault tolerance must be asymmetric.

Why liveness must not check DB? DB blip → full cluster restart → local fault amplified.

How to use SmartLifecycle phase? Higher phase stops first; same phase concurrent; used to orchestrate destruction order.

What if stop(Runnable) doesn't call callback? Whole phase group waits timeout, grace period wasted.

@PreDestroy vs SmartLifecycle who runs first? SmartLifecycle first (onClose before destroyBeans).

Container close five steps? Event → onClose → destroyBeans → clear caches → close BeanFactory.

How to calculate terminationGracePeriod? preStop + grace + probe latency + margin.

What is OUT_OF_SERVICE for? Drop traffic without triggering restart; standard active drain posture.

Why multiple shutdown hooks dangerous? JDK runs concurrently, no order guarantee.

How to locate "shutdown slow"? Enable DefaultLifecycleProcessor DEBUG, watch each phase duration.

10. One-Page Cheat Sheet

[Essence] Pause accept + drain in-flight
Not sleep, poll counters
[Two switches] server.shutdown=graceful
lifecycle.timeout-per-shutdown
[Five close steps] Event → onClose
→ destroyBeans → clear caches
→ close BeanFactory
[Phase rule] Higher first, same phase concurrent
Web container = MAX_VALUE
[Callback] stop(Runnable) MUST call
Else whole group waits timeout
[Time math] preStop + grace + probe
+ margin < gracePeriod
[Three probes] startup waits
liveness restarts
readiness drops traffic
[Iron law] Liveness no DB
Dependency blip don't amplify to restart
[Three states] UP / DOWN / OUT_OF_SERVICE
Last one = drop traffic no restart
[Groups] probes.enabled=true
liveness / readiness separated
[Active drain] Flip readiness → wait
→ then exit, beats guessing time
[Debug] DefaultLifecycleProcessor
DEBUG to see phase durations

Shutdown and health checks share a trait: "They only matter in rare moments, and those moments are exactly the most dangerous ones." Day-to-day no difference; once you deploy, scale, or a dependency blips, the gap equals hundreds of user orders.

Don't treat it as "ops work." An application knowing how to die gracefully and report health honestly is basic service hygiene — like a professional knowing how to hand over work properly, not just walk away.

If you take one sentence: Graceful shutdown's core isn't "wait," it's "let traffic leave first, let requests finish first, let dependencies close last." Get those three precedence relationships right, and config parameters fall into place.