Zero Downtime Isn't Accidental: Deep Dive into Kubernetes Smooth Deployments from Theory to Production

Why Rolling Updates Still Cause Jitter

Typical rollout incidents are not full cluster crashes but short periods (seconds to dozens of seconds) of service jitter. Common symptoms include:

New pods become Ready but request latency spikes and timeouts appear.

Old pods stay Terminating while still receiving traffic.

Ingress connection reuse continues to send traffic to pods that are about to exit.

Service‑registry caches are not refreshed, so consumers still call the old instance.

Kafka/RocketMQ consumers are abruptly stopped, triggering a rebalance storm.

HPA scaling during the rollout causes version mixing, hotspot skew, and capacity mis‑judgment.

The root cause is that the control plane believes the node has switched, while the data‑plane traffic has not fully migrated.

Zero‑downtime therefore depends on a set of engineering constraints rather than merely enabling strategy.type=RollingUpdate:

New instances must not receive traffic before they are truly ready.

Old instances must not be killed before they have drained all in‑flight requests.

All entry layers, service‑discovery layers, and application layers must agree on the "online" and "offline" state.

Capacity during the rollout must stay above the real‑time load plus a safety margin.

Any anomaly must be automatically stopped and rolled back within minutes.

Real‑World Fault Timeline

Assume an order‑service upgrading from v1 to v2 with 8 pods, a flat‑peak of 3k QPS, a high‑peak of 12k QPS, and a per‑pod capacity of ~2k QPS.

Deployment creates a new pod using RollingUpdate.

New pod starts quickly, HTTP port opens, readiness probe passes after 5 seconds.

Service starts routing traffic to the new pod.

JVM just finished start‑up; thread pool, connection pool, Redis hot‑keys, and rule caches are still warming up, causing request latency to jump from 20 ms to 800 ms.

Old pod is marked Terminating and removed from EndpointSlice.

kube‑proxy, Ingress, and client connection pools still see the old endpoint for a short period.

Application receives SIGTERM and exits immediately, interrupting in‑flight requests, resetting long connections, and leaving Kafka offsets uncommitted.

Monitoring shows a brief but obvious error spike: HTTP 5xx, RPC timeouts, duplicate MQ consumption, and Nacos instance jitter.

This illustrates the classic "fake smooth rollout": control‑plane appears smooth, data‑plane does not.

The Essence of Zero‑Downtime: Four Key Sequences

3.1 Pod Termination Sequence

When a pod is deleted the following steps occur:

API Server sets deletionTimestamp.

EndpointSlice controller removes the pod from the Service backend list.

kube‑proxy synchronises iptables/ipvs rules.

Ingress controller / Service mesh / cloud LB perceives the backend change.

Connection pools, HTTP/2, gRPC streams gradually drain.

kubelet runs the preStop hook.

Container receives SIGTERM.

If the pod does not exit after terminationGracePeriodSeconds, SIGKILL is sent.

The hidden propagation delay between "removed from Service" and "no more requests arrive" means preStop + sleep 10 is never sufficient; it only gives time for the delay to settle.

3.2 Pod Readiness Sequence

Readiness probe passing does not guarantee the pod can handle production traffic. Real readiness must cover three layers:

Process layer – the process is alive and the port is listening.

Dependency layer – databases, caches, config centre, message systems are reachable.

Business layer – critical resources are warmed up and the instance can safely receive traffic.

Typical warm‑up steps include Spring lazy‑bean init, DB pool fill, Redis hot‑key pre‑heat, rule cache load, consumer group rebalance, service‑registry registration, and JIT compilation.

3.3 Capacity Sequence

During a rolling update the cluster capacity changes dynamically. Two common risks are: maxUnavailable too large – old pods disappear first, capacity drops below the safety line. maxSurge too small – new pods start slowly, leaving insufficient redundancy during peak load.

Zero‑downtime means the total effective capacity stays above the real‑time load plus the safety margin throughout the rollout.

3.4 Rollback Sequence

Manual Grafana observation and manual rollback are insufficient for high‑frequency releases. A reliable approach is:

Observe during the rollout.

Pause on failure.

Automatically trigger rollback when metrics cross thresholds.

Rollback should be a built‑in branch of the release system, not a manual remediation step.

Production‑Ready Architecture (simplified)

┌──────────────────────────────┐
                │          Client / App          │
                └──────────────┬───────────────┘
                               │
                         ┌─────▼─────┐
                         │ Ingress / │
                         │ Gateway   │
                         │ Nginx/Env │
                         └─────┬─────┘
                               │
                         ┌─────▼─────┐
                         │ Service / │
                         │ DNS VIP   │
                         └─────┬─────┘
                               │
          ┌─────────────────────┼─────────────────────┐
          │                     │                     │
   ┌──────▼───────┐   ┌───────▼───────┐   ┌───────▼───────┐
   │ Pod A (old)  │   │ Pod B (old)   │   │ Pod C (new)   │
   │ ready=false │   │ draining     │   │ warmup        │
   └──────┬───────┘   └───────┬───────┘   └───────┬───────┘
          │               │               │
   ┌──────▼─────┐ ┌───────▼─────┐ ┌─────▼─────┐
   │ offline    │ │ inflight    │ │ preload   │
   │ endpoint   │ │ drain       │ │ caches    │
   └──────┬─────┘ └───────┬─────┘ └───────┬─────┘
          │               │               │
   ┌──────▼─────┐ ┌───────▼─────┐ ┌─────▼─────┐
   │ Nacos      │ │ MQ consumer │ │ readiness │
   │ deregister │ │ stop        │ │ open      │
   └──────┬─────┘ └───────┬─────┘ └───────┬─────┘
          │               │               │
          └───────┬───────┴───────┬───────┘
                  │               │
               ┌──▼─────┐   ┌─────▼─────┐
               │Prometheus│ │Rollout    │
               │metrics   │ │Engine    │
               └──────────┘ └──────────┘

Design principle: pre‑heat before scaling up, deregister before draining, and only exit after all in‑flight requests have finished.

Kubernetes‑Level Production Settings

5.1 Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: prod
  labels:
    app: order-service
spec:
  replicas: 8
  revisionHistoryLimit: 5
  minReadySeconds: 20
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      terminationGracePeriodSeconds: 90
      containers:
      - name: order-service
        image: registry.example.com/order-service:2.4.3
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 8080
        env:
        - name: JAVA_OPTS
          value: "-XX:+UseG1GC -XX:MaxRAMPercentage=70"
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - |
                curl -fsS -X POST http://127.0.0.1:8080/internal/offline || true
                sleep 20
        startupProbe:
          httpGet:
            path: /actuator/health/startup
            port: 8080
          periodSeconds: 5
          failureThreshold: 24
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          periodSeconds: 10
        resources:
          requests:
            cpu: "1000m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "2Gi"

Key differences from default examples: maxUnavailable: 0 guarantees old pods stay alive until new ones are ready. maxSurge: 25% allows temporary capacity expansion for warm‑up. minReadySeconds: 20 forces a 20‑second stability window after readiness probe passes. startupProbe prevents a liveness kill before the application finishes cold start. terminationGracePeriodSeconds: 90 covers the whole "offline → propagation delay → request drain → MQ close → process exit" window.

5.2 PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-service-pdb
  namespace: prod
spec:
  minAvailable: 7
  selector:
    matchLabels:
      app: order-service

With 8 replicas, at most one pod may be voluntarily evicted, protecting capacity during node maintenance or Spot‑instance reclamation.

5.3 Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
  namespace: prod
spec:
  minReplicas: 8
  maxReplicas: 20
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 20
        periodSeconds: 60

Key ideas:

Set minReplicas to the current safe capacity before the rollout to prevent HPA from shrinking during the rollout.

Add a stabilization window on scale‑down to avoid reacting to transient spikes caused by warm‑up.

Pre‑scale in the pipeline (e.g., add 20‑50% extra replicas) for high‑peak releases.

Application‑Level Smooth Deployment

6.1 State Machine

Define four explicit states in the application:

public enum State {
    STARTING,
    READY,
    DRAINING,
    STOPPED
}

Transition logic ensures traffic is only allowed in READY and rejected in DRAINING. This is more reliable than relying on Spring Boot's default graceful shutdown.

6.2 TrafficGate Component (Java)

package com.example.orders.deploy;

import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;

@Component
public class TrafficGate {
    public enum State { STARTING, READY, DRAINING, STOPPED }
    private final AtomicReference<State> state = new AtomicReference<>(State.STARTING);
    private final AtomicInteger inflight = new AtomicInteger(0);

    public TrafficGate(MeterRegistry registry) {
        Gauge.builder("app_inflight_requests", inflight, AtomicInteger::get).register(registry);
        Gauge.builder("app_traffic_state", state, s -> s.get().ordinal()).register(registry);
    }
    public boolean allowTraffic() { return state.get() == State.READY; }
    public boolean enterReady() { return state.compareAndSet(State.STARTING, State.READY); }
    public boolean enterDraining() {
        State cur = state.get();
        if (cur == State.DRAINING || cur == State.STOPPED) return false;
        return state.getAndSet(State.DRAINING) != State.DRAINING;
    }
    public void incrementInflight() { inflight.incrementAndGet(); }
    public void decrementInflight() { inflight.decrementAndGet(); }
    public boolean awaitInflightZero(long timeout, TimeUnit unit) throws InterruptedException {
        long deadline = System.nanoTime() + unit.toNanos(timeout);
        while (System.nanoTime() < deadline) {
            if (inflight.get() == 0) { state.set(State.STOPPED); return true; }
            Thread.sleep(200);
        }
        return inflight.get() == 0;
    }
    public State currentState() { return state.get(); }
}

6.3 TrafficGateFilter (Servlet Filter)

package com.example.orders.deploy;

import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import java.io.IOException;

@Component
@Order(Ordered.HIGHEST_PRECEDENCE)
public class TrafficGateFilter extends OncePerRequestFilter {
    private final TrafficGate trafficGate;
    public TrafficGateFilter(TrafficGate trafficGate) { this.trafficGate = trafficGate; }
    @Override
    protected boolean shouldNotFilter(HttpServletRequest request) {
        String uri = request.getRequestURI();
        return uri.startsWith("/actuator") || uri.startsWith("/internal");
    }
    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        if (!trafficGate.allowTraffic()) {
            response.setStatus(HttpStatus.SERVICE_UNAVAILABLE.value());
            response.getWriter().write("instance is draining");
            return;
        }
        trafficGate.incrementInflight();
        try { chain.doFilter(request, response); }
        finally { trafficGate.decrementInflight(); }
    }
}

6.4 Readiness Health Indicator

package com.example.orders.deploy;

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component("trafficGate")
public class TrafficGateHealthIndicator implements HealthIndicator {
    private final TrafficGate trafficGate;
    private final DependencyWarmupService warmupService;
    public TrafficGateHealthIndicator(TrafficGate trafficGate, DependencyWarmupService warmupService) {
        this.trafficGate = trafficGate;
        this.warmupService = warmupService;
    }
    @Override
    public Health health() {
        boolean ready = trafficGate.currentState() == TrafficGate.State.READY && warmupService.isReady();
        if (ready) return Health.up().withDetail("state", "READY").build();
        return Health.outOfService()
                .withDetail("state", trafficGate.currentState().name())
                .withDetail("warmupReady", warmupService.isReady())
                .build();
    }
}

6.5 Dependency Warm‑up Service

package com.example.orders.deploy;

import org.springframework.stereotype.Service;
import jakarta.annotation.PostConstruct;
import java.util.concurrent.atomic.AtomicBoolean;

@Service
public class DependencyWarmupService {
    private final AtomicBoolean ready = new AtomicBoolean(false);
    private final RuleCache ruleCache;
    private final PriceService priceService;
    private final InventoryGateway inventoryGateway;
    public DependencyWarmupService(RuleCache ruleCache, PriceService priceService, InventoryGateway inventoryGateway) {
        this.ruleCache = ruleCache;
        this.priceService = priceService;
        this.inventoryGateway = inventoryGateway;
    }
    @PostConstruct
    public void warmup() {
        ruleCache.preload();
        priceService.preloadHotSkuPrices();
        inventoryGateway.ping();
        ready.set(true);
    }
    public boolean isReady() { return ready.get(); }
}

6.6 Ready State Listener

package com.example.orders.deploy;

import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

@Component
public class ReadyStateListener {
    private final TrafficGate trafficGate;
    private final DependencyWarmupService warmupService;
    public ReadyStateListener(TrafficGate trafficGate, DependencyWarmupService warmupService) {
        this.trafficGate = trafficGate;
        this.warmupService = warmupService;
    }
    @EventListener(ApplicationReadyEvent.class)
    public void onApplicationReady() {
        if (warmupService.isReady()) {
            trafficGate.enterReady();
        }
    }
}

6.7 Offline Controller (preStop entry point)

package com.example.orders.deploy;

import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/internal")
public class OfflineController {
    private final GracefulShutdownCoordinator coordinator;
    public OfflineController(GracefulShutdownCoordinator coordinator) { this.coordinator = coordinator; }
    @PostMapping("/offline")
    public ResponseEntity<String> offline() {
        coordinator.beginDraining();
        return ResponseEntity.ok("draining");
    }
}

6.8 Graceful Shutdown Coordinator

package com.example.orders.deploy;

import jakarta.annotation.PreDestroy;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import java.util.concurrent.TimeUnit;

@Component
public class GracefulShutdownCoordinator {
    private static final Logger log = LoggerFactory.getLogger(GracefulShutdownCoordinator.class);
    private final TrafficGate trafficGate;
    private final ServiceRegistryAdapter serviceRegistryAdapter;
    private final MessageConsumerCoordinator messageConsumerCoordinator;
    public GracefulShutdownCoordinator(TrafficGate trafficGate, ServiceRegistryAdapter serviceRegistryAdapter,
                                      MessageConsumerCoordinator messageConsumerCoordinator) {
        this.trafficGate = trafficGate;
        this.serviceRegistryAdapter = serviceRegistryAdapter;
        this.messageConsumerCoordinator = messageConsumerCoordinator;
    }
    public synchronized void beginDraining() {
        if (!trafficGate.enterDraining()) return;
        log.info("instance entering draining state");
        serviceRegistryAdapter.deregister();
        messageConsumerCoordinator.pause();
    }
    @PreDestroy
    public void shutdown() throws InterruptedException {
        beginDraining();
        boolean drained = trafficGate.awaitInflightZero(45, TimeUnit.SECONDS);
        messageConsumerCoordinator.shutdown();
        log.info("graceful shutdown finished, drained={}", drained);
    }
}

6.9 Nacos Deregistration Adapter

package com.example.orders.deploy;

import com.alibaba.cloud.nacos.registry.NacosRegistration;
import com.alibaba.cloud.nacos.NacosServiceManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;

@Component
public class NacosServiceRegistryAdapter implements ServiceRegistryAdapter {
    private static final Logger log = LoggerFactory.getLogger(NacosServiceRegistryAdapter.class);
    private final NacosRegistration registration;
    private final NacosServiceManager nacosServiceManager;
    public NacosServiceRegistryAdapter(NacosRegistration registration, NacosServiceManager nacosServiceManager) {
        this.registration = registration;
        this.nacosServiceManager = nacosServiceManager;
    }
    @Override
    public void deregister() {
        try {
            nacosServiceManager.getNamingService().deregisterInstance(
                registration.getServiceId(), registration.getGroup(), registration.getHost(), registration.getPort());
            log.info("nacos deregistered");
        } catch (Exception e) {
            log.warn("failed to deregister from nacos", e);
        }
    }
}

6.10 Kafka Consumer Coordinator

package com.example.orders.deploy;

import org.springframework.kafka.config.KafkaListenerEndpointRegistry;
import org.springframework.kafka.listener.MessageListenerContainer;
import org.springframework.stereotype.Component;

@Component
public class KafkaConsumerCoordinator implements MessageConsumerCoordinator {
    private final KafkaListenerEndpointRegistry registry;
    public KafkaConsumerCoordinator(KafkaListenerEndpointRegistry registry) { this.registry = registry; }
    @Override
    public void pause() { registry.getListenerContainers().forEach(MessageListenerContainer::pause); }
    @Override
    public void shutdown() { registry.getListenerContainers().forEach(MessageListenerContainer::stop); }
}

For native KafkaConsumer the safe shutdown sequence is:

Call pause(assignment()).

Wait for in‑flight messages to finish.

Commit offsets with commitSync().

Wake up the consumer with wakeup().

Finally close the consumer.

High‑Concurrency Pitfalls

Long‑Living Connections

HTTP Keep‑Alive, HTTP/2, gRPC streams, and WebSocket connections continue to send traffic to a pod even after it is removed from the Service list. The connection is only closed when the client disconnects, the server gracefully shuts down, or the proxy drains. Ingress Nginx can be configured to drain connections:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  worker-shutdown-timeout: "120s"
  keep-alive: "75"
  keep-alive-requests: "1000"
  upstream-keepalive-connections: "64"
  upstream-keepalive-timeout: "60"

For Envoy/Istio the relevant fields are drainDuration, terminationDrainDuration, and upstream connection‑pool settings.

Capacity Planning for Peak Traffic

Example: current QPS 12 000, per‑pod safe capacity 2 000, 8 pods → theoretical 16 000 QPS. With a 20 % safety margin the safe line is 14 400 QPS. During a rollout the effective capacity must never drop below this value, otherwise latency and error rates will rise even without bugs.

Database Compatibility (Expand‑Contract)

Zero‑downtime requires forward‑compatible schema changes:

Add new columns, indexes, or tables without dropping old ones.

Make application code read/write both old and new formats.

After all pods run the new version and are stable, clean up old schema elements.

Common anti‑patterns include writing new fields that old code cannot read, dropping columns before all pods have upgraded, or locking tables with DDL that spikes latency.

HPA Interaction with Cold Pods

Cold pods receiving traffic cause CPU spikes, Full GC, latency spikes, and HPA over‑scaling, creating a feedback loop. Mitigations:

Pre‑scale before the rollout.

Use minReadySeconds to keep pods in a warm state before they accept traffic.

Canary rollout with incremental weights (1 %, 5 %, 10 %, 25 %, 50 %, 100 %).

Run warm‑up load tests after container start.

Canary / Blue‑Green Strategies

Argo Rollouts Example

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
  namespace: prod
spec:
  replicas: 8
  strategy:
    canary:
      maxSurge: 25%
      maxUnavailable: 0
      steps:
      - setWeight: 5
      - pause:
          duration: 3m
      - setWeight: 20
      - pause:
          duration: 5m
      - setWeight: 50
      - pause:
          duration: 10m
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: registry.example.com/order-service:2.4.3

Metrics such as new‑version 5xx ratio, P95/P99 latency, JVM GC count, Kafka lag, DB slow‑SQL count, and Nacos instance jitter are monitored. If any metric exceeds a threshold, the rollout is automatically paused or rolled back.

Monitoring, Alerting, and Rollback Loop

Essential Metrics (converted from table)

Traffic Quality : HTTP 5xx, RPC error rate – detect error spikes.

Latency : P95/P99 response time – watch tail latency degradation.

Instance State : Ready pod count, restart count – ensure capacity stays stable.

JVM : CPU, heap usage, GC count – detect cold‑start resource spikes.

Service Discovery : Nacos instance change frequency – watch frequent up/down events.

MQ : Consumer lag, rebalance count – check consumption impact.

Ingress : 502/503, upstream reset – verify smooth connection draining.

Alert Rules for Release Windows

New version 5xx rate exceeds old version by 0.5 % in a 5‑minute window.

P99 latency exceeds baseline by 30 %.

Ready pod count falls below the minimum safe replica count.

Kafka lag grows continuously for 3 minutes.

Container OOM events are non‑zero.

Automated Rollback Triggers in CI/CD

Pre‑release checks: recent error rate, latency, lag, and DB slow‑SQL are normal.

Observe during each canary step with a 3‑10 minute observation window.

Post‑release validation: core metrics return to baseline before closing the window.

Automatic stop‑loss: any metric crossing its threshold pauses or rolls back the rollout.

Production Case Study: Order Service

Background : 8‑pod order service, flat‑peak 4k QPS, high‑peak 12k QPS, Spring Boot + Nacos + Kafka + MySQL + Redis, deployed via Kubernetes + Ingress Nginx. Each release previously produced 10‑30 seconds of 502/503 spikes.

Problems Identified

Readiness probe only checked port. preStop had no logic.

Spring Boot exited immediately on SIGTERM.

Nacos deregistration relied on heartbeat timeout.

Kafka consumer called stop() directly.

No pre‑scale for peak‑time releases.

Remediation Steps

Add startupProbe, readinessProbe, and minReadySeconds.

Implement application‑level state machine (STARTING/READY/DRAINING/STOPPED).

Make preStop invoke /internal/offline which triggers draining.

Active Nacos deregistration during DRAINING.

Replace Kafka stop() with pause → drain → stop sequence.

Pre‑scale replicas from 8 to 10 before the rollout.

Switch rollout strategy to canary 5 % → 20 % → 50 % → 100 %.

Results

First‑batch P99 latency dropped from 1.4 s to 220 ms.

HTTP 5xx peak reduced from 0.8 % to 0 %.

Kafka duplicate‑consume alerts disappeared.

Ready pod count never fell below the safety line.

Manual monitoring reduced to only watching pipeline alerts.

Top 10 Common Pitfalls

Assuming RollingUpdate equals zero‑downtime.

Believing preStop sleep 30 is enough.

Thinking readiness probe passing means the service is stable.

Relying on framework‑provided graceful shutdown without handling MQ, service‑registry, or cache.

Ignoring database changes, connection pools, and long‑lived streams.

Avoiding releases during peak traffic without capacity planning.

Assuming rollback is always safer than continuing.

Setting HPA too sensitive, causing feedback loops.

Monitoring only the application layer and forgetting ingress, service‑discovery, MQ, DB, or node health.

Treating zero‑downtime as a one‑time configuration instead of an evolving process.

Executable Release Checklist

Before Release

Core services show normal error rate, latency, and MQ lag for the last 30 minutes.

Current replica count satisfies peak‑load safety margin.

Database migrations follow Expand‑Contract.

New version start‑up time and resource usage have been validated.

Readiness/startup/liveness probes are calibrated.

PodDisruptionBudget, HPA, anti‑affinity/topologySpread constraints are in place.

Rollback image version is still available.

During Release

New instances pre‑heat before receiving traffic.

Old instances transition to DRAINING before exiting.

Ready pod count never drops below the safety threshold.

Continuously compare new vs old version latency, error rate, GC, and lag.

Each traffic‑increase step has a sufficient observation window.

After Release

New version metrics stabilize back to baseline.

Nacos, Service, and Ingress backend lists are consistent.

No zombie connections, duplicate consumption, or slow‑SQL spikes.

Observe for 10‑30 minutes before closing the release window.

Conclusion

Zero‑downtime releases are not a Kubernetes trick or a handful of YAML tweaks. They require coordinated effort across the control plane, service mesh, application code, service registry, message system, database, and monitoring pipeline. When all pieces form a closed loop, releases become repeatable, verifiable, roll‑back‑able, and scalable engineering capabilities.