Industrial‑Grade Kubernetes Deployment: Zero‑Downtime Rolling Updates for Million‑Scale Concurrency

Introduction: A Real‑World Incident

During a low‑risk version upgrade (v2.3.7 → v2.3.8) of an e‑commerce order service, three alerts fired 40 seconds after rollout started: P99 latency jumped from 180 ms to 4.8 s, 5xx error rate rose from 0.03% to 7.4%, and Kafka consumer lag surged from 20 k to 800 k messages.

Post‑mortem revealed the root cause was an incomplete deployment pipeline, not a code bug. Five specific issues were identified:

New Pods were added to the Service before connection pools, caches, and downstream authentication were ready.

The maxUnavailable=50% setting removed half of the available capacity instantly.

On receiving SIGTERM, the application did not stop traffic, pause consumption, or finish in‑flight requests, causing double loss of requests and messages.

HPA scaled only on CPU, ignoring QPS and queue backlog, so capacity lagged behind traffic spikes.

Images were tagged :latest, making rollbacks impossible because the previous stable image could not be identified.

These problems are typical in production Kubernetes environments where teams use Deployments without treating them as full‑featured delivery controllers.

Why Deployment Is the Core of Production Release Control

Deployment does more than start Pods; it maintains a desired state declaratively. In the classic controller diagram:

CI/CD → Image Registry → Deployment → ReplicaSet → Pod
                                   ↘︎ Service ↔︎ EndpointSlice
                                   ↘︎ Ingress/Gateway
Pod → ConfigMap/Secret, Probe, Metrics, PDB, Affinity, Kafka/Redis/DB, Tracing

In this model, the application code decides whether the business can run, while Deployment decides whether the business can run stably, controllably, and with low risk .

Workloads Suitable for Deployment

Web API services

BFF / Gateway back‑ends

Stateless order, inventory, payment, search services

Message consumer services

Mid‑tier aggregation services

AI inference gateways, retrieval services, feature services

Stateful components such as MySQL, PostgreSQL, Redis master‑slave, Kafka, ZooKeeper, etc., should use StatefulSet instead.

Deployment Mechanics and the Reconcile Loop

When the spec.template changes, the controller creates a new ReplicaSet, scales it up according to the update strategy, scales the old ReplicaSet down, and finally marks the version switch complete. This design provides:

Automatic history retention (old ReplicaSets remain until explicitly removed).

Rollback by re‑scaling the old ReplicaSet.

Clear intermediate states for easier troubleshooting.

The underlying reconcile loop follows Observe → Diff → Act repeatedly until the actual state matches the desired state.

Full Rolling‑Update Timeline

1. User updates Deployment image version
2. Controller detects PodTemplate change
3. New ReplicaSet is created
4. New Pods are added according to maxSurge
5. Scheduler assigns nodes
6. kubelet pulls image and creates containers
7. startupProbe / readinessProbe run
8. Once Ready, EndpointSlice is updated
9. kube‑proxy / CNI updates forwarding tables
10. Service starts sending traffic to new Pods
11. Controller reduces old ReplicaSet per maxUnavailable
12. Old Pods receive SIGTERM, execute preStop, gracefully exit
13. Steps repeat until all traffic is on the new ReplicaSet

Steps 8–12 are often overlooked; propagation delays between the control plane (EndpointSlice) and data plane (kube‑proxy) can cause traffic to reach Pods that are not truly ready.

Design Principles for Zero‑Downtime Deployments

Traffic must be admitted only after the application is truly ready. Readiness should verify configuration loading, thread‑pool initialization, DB connectivity, Redis access, Kafka producer/consumer health, service‑registry registration, and cache warm‑up.

Graceful shutdown must follow a strict order:

Reject new traffic.

Stop consuming new messages.

Wait for in‑flight requests to finish.

Flush caches, logs, and tracing data.

Close thread‑pools and connections.

Terminate the process.

Release strategy must be capacity‑aware. Compute the required replica count from single‑Pod capacity, peak traffic, and a safety factor (e.g., 1.5×). For a target of 12 000 QPS with a single‑Pod stable QPS of 800, the calculation yields at least 22‑23 replicas; round up to 24‑28 for redundancy.

High availability requires fault‑domain isolation. Use topologySpreadConstraints and podAntiAffinity to spread Pods across zones, nodes, and hostnames, and set PodDisruptionBudget with a realistic minAvailable value.

Observability must be baked into the Deployment. Include detailed health endpoints (startup, liveness, readiness) and expose dependency health (DB, Redis, Kafka) in the readiness response.

Key Configuration Examples

Full Production‑Grade Deployment YAML (order‑service)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
  labels:
    app.kubernetes.io/name: order-service
    app.kubernetes.io/part-of: ecommerce
    app.kubernetes.io/component: trade
    app.kubernetes.io/version: "2.3.8"
    app.kubernetes.io/managed-by: argocd
  annotations:
    kubernetes.io/change-cause: "release 2.3.8 fix idempotent conflict and optimize inventory timeout"
spec:
  replicas: 24
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  minReadySeconds: 20
  selector:
    matchLabels:
      app.kubernetes.io/name: order-service
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order-service
        app.kubernetes.io/part-of: ecommerce
        app.kubernetes.io/component: trade
        app.kubernetes.io/version: "2.3.8"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      serviceAccountName: order-service-sa
      priorityClassName: online-critical
      terminationGracePeriodSeconds: 90
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: order-service
        - maxSkew: 2
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: order-service
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: order-service
                topologyKey: kubernetes.io/hostname
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: order-service
          image: registry.example.com/ecommerce/order-service:2.3.8@sha256:7f3a2f3af0d51234abcd5678ef901234567890abcdef1234567890abcdef12
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: prod
            - name: TZ
              value: Asia/Shanghai
            - name: JAVA_TOOL_OPTIONS
              value: >-
                -XX:+UseContainerSupport
                -XX:InitialRAMPercentage=50
                -XX:MaxRAMPercentage=70
                -XX:+HeapDumpOnOutOfMemoryError
                -XX:HeapDumpPath=/tmp
                -XX:+ExitOnOutOfMemoryError
            - name: NACOS_SERVER_ADDR
              value: nacos-headless.nacos.svc.cluster.local:8848
            - name: DB_URL
              valueFrom:
                secretKeyRef:
                  name: order-service-secret
                  key: db-url
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: order-service-secret
                  key: db-username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: order-service-secret
                  key: db-password
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: order-service-secret
                  key: redis-password
          resources:
            requests:
              cpu: "2000m"
              memory: "4Gi"
            limits:
              cpu: "4000m"
              memory: "4Gi"
          startupProbe:
            httpGet:
              path: /actuator/health/startup
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
            successThreshold: 2
          lifecycle:
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - >-
                    wget -qO- http://127.0.0.1:8080/actuator/service-registry/offline || true;
                    wget -qO- -S --post-data='' http://127.0.0.1:8080/actuator/order/drain || true;
                    sleep 25
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: app-config
              mountPath: /app/config
              readOnly: true
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: app-config
          configMap:
            name: order-service-config
        - name: tmp
          emptyDir:
            sizeLimit: 2Gi

Service Definition

apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: production
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: order-service
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
  sessionAffinity: None

PodDisruptionBudget

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-service-pdb
  namespace: production
spec:
  minAvailable: 18
  selector:
    matchLabels:
      app.kubernetes.io/name: order-service

Horizontal Pod Autoscaler (CPU + QPS + Kafka Lag)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 24
  maxReplicas: 80
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "650"
    - type: Pods
      pods:
        metric:
          name: kafka_consumer_lag
        target:
          type: AverageValue
          averageValue: "3000"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 12
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      selectPolicy: Min
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Readiness Health Indicator (Spring Boot)

package com.company.order.health;

import java.time.Duration;
import java.util.concurrent.atomic.AtomicBoolean;
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.kafka.core.KafkaAdmin;
import org.springframework.stereotype.Component;

@Component("readiness")
public class ReadinessHealthIndicator implements HealthIndicator {
    private final JdbcTemplate jdbcTemplate;
    private final StringRedisTemplate redisTemplate;
    private final KafkaAdmin kafkaAdmin;
    private final AtomicBoolean draining = new AtomicBoolean(false);

    public ReadinessHealthIndicator(JdbcTemplate jdbcTemplate, StringRedisTemplate redisTemplate, KafkaAdmin kafkaAdmin) {
        this.jdbcTemplate = jdbcTemplate;
        this.redisTemplate = redisTemplate;
        this.kafkaAdmin = kafkaAdmin;
    }

    @Override
    public Health health() {
        if (draining.get()) {
            return Health.outOfService()
                .withDetail("reason", "instance is draining")
                .build();
        }
        try {
            jdbcTemplate.queryForObject("select 1", Integer.class);
            redisTemplate.getConnectionFactory().getConnection().ping();
            kafkaAdmin.describeTopics("order-created");
            return Health.up()
                .withDetail("db", "ok")
                .withDetail("redis", "ok")
                .withDetail("kafka", "ok")
                .build();
        } catch (Exception ex) {
            return Health.outOfService()
                .withDetail("reason", ex.getMessage())
                .build();
        }
    }

    public void markDraining() {
        draining.set(true);
        try {
            Thread.sleep(Duration.ofSeconds(20));
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }
}

Drain Controller (exposes the preStop endpoint)

package com.company.order.admin;

import com.company.order.health.ReadinessHealthIndicator;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DrainController {
    private final ReadinessHealthIndicator readinessHealthIndicator;

    public DrainController(ReadinessHealthIndicator readinessHealthIndicator) {
        this.readinessHealthIndicator = readinessHealthIndicator;
    }

    @PostMapping("/actuator/order/drain")
    public ResponseEntity<Void> drain() {
        readinessHealthIndicator.markDraining();
        return ResponseEntity.accepted().build();
    }
}

Consumer Shutdown Coordinator (Kafka consumer graceful stop)

package com.company.order.messaging;

import jakarta.annotation.PreDestroy;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.config.KafkaListenerEndpointRegistry;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Component;

@Component
public class ConsumerShutdownCoordinator {
    private static final Logger log = LoggerFactory.getLogger(ConsumerShutdownCoordinator.class);
    private final KafkaListenerEndpointRegistry registry;
    private final ThreadPoolTaskExecutor orderExecutor;

    public ConsumerShutdownCoordinator(KafkaListenerEndpointRegistry registry, ThreadPoolTaskExecutor orderExecutor) {
        this.registry = registry;
        this.orderExecutor = orderExecutor;
    }

    @PreDestroy
    public void shutdown() {
        log.info("order-service is shutting down, stop kafka listeners first");
        registry.stop();
        orderExecutor.shutdown();
        try {
            if (!orderExecutor.getThreadPoolExecutor().awaitTermination(30, TimeUnit.SECONDS)) {
                log.warn("executor did not terminate in time");
            }
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }
}

Progressive Delivery and Canary Strategies

For traffic‑percentage rollouts, an NGINX Ingress canary be used:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-service-canary
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service-canary
                port:
                  number: 80

Argo Rollouts enables fully automated canary analysis and rollback:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 24
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: registry.example.com/ecommerce/order-service:2.3.9
  strategy:
    canary:
      maxSurge: 25%
      maxUnavailable: 0
      canaryService: order-service-canary
      stableService: order-service-stable
      trafficRouting:
        nginx:
          stableIngress: order-service
      steps:
        - setWeight: 5
        - pause: {duration: 5m}
        - setWeight: 20
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: order-success-rate
        - setWeight: 50
        - pause: {duration: 10m}
        - setWeight: 100

GitOps‑Driven Delivery Pipeline

A typical production pipeline looks like:

Code commit → CI compile & test → Build image + SBOM → Security scan → Push image → Update Helm/Kustomize manifests → PR to GitOps repo → Code review → Argo CD/Flux sync → Rollout/Deployment → Prometheus metric validation → Auto‑scale or auto‑rollback

Git becomes the single source of truth; every change is versioned, auditable, and reproducible.

Common Pitfalls and Troubleshooting

Top 10 high‑frequency traps include missing or optimistic readiness probes, aggressive liveness probes, oversized maxUnavailable, lack of graceful shutdown, CPU‑only HPA, overly strict anti‑affinity, large images, missing PDB, configuration‑only rollbacks, and deploying to the default namespace.

Fast‑track troubleshooting table (symptom → likely cause → key command) is provided in the original article and should be kept as a reference during incidents.

Release Checklist

Verify image tag and digest.

Confirm readinessProbe reflects real business readiness.

Ensure maxUnavailable=0 for critical services.

Check that a PodDisruptionBudget exists.

Validate HPA minimum replicas cover peak load.

Confirm configuration compatibility.

Make sure rollback image is still available.

Verify database migrations are reversible or backward‑compatible.

Post‑Release Observation (first 5‑15 min)

New vs. old Pod counts.

Error rates.

P95/P99 latency.

CPU / memory usage.

Full GC occurrences.

DB connection pool saturation and slow queries.

Redis hit‑rate.

Kafka lag.

Core business success rates (order creation, payment callbacks, etc.).

Evolution Roadmap for Teams

Stage 1 – Basic Availability: Containerize services, use simple Deployment + Service, add basic probes.

Stage 2 – Release Safety: Set maxUnavailable=0, add graceful shutdown, PDB, resource requests/limits, versioned images.

Stage 3 – Elastic Capacity: Introduce HPA with multi‑metric policies, node‑pool scaling, pre‑capacity planning for peaks.

Stage 4 – Release Governance: Adopt GitOps, implement canary/gray‑release, automated metric gates, auto‑rollback.

Stage 5 – Multi‑Cluster & Disaster Recovery: Deploy across zones/regions, synchronize traffic, ensure cross‑cluster failover.

Conclusion

Deployment is far more than a replica controller; it is the engineering method that turns business code into a continuously stable, observable, and controllable production service. Mastering probes, update strategies, capacity modeling, fault‑domain isolation, and GitOps‑driven governance transforms a simple YAML file into a robust, zero‑downtime delivery pipeline capable of supporting million‑scale concurrent traffic.