Kubernetes Rolling Updates & Rollbacks: From Deployment Mechanics to Release

Why many teams can use Deployment but cannot do production releases

Teams often view a rolling update as three steps: change the image, wait for the new pod to become Ready, then delete the old pod. This view ignores that Kubernetes only reconciles replica counts; it does not guarantee business correctness. Production‑grade rolling updates must consider whether a new pod can handle load, whether old pods still have in‑flight requests, resource saturation, HPA/PDB interactions, rollback image availability, and traffic‑mesh convergence.

Typical incident: "Pod is Ready" but the service still fails

A trading platform order service runs 12 replicas (≈4 k TPS, peak 25 k TPS). A release changed connection‑pool and thread‑pool parameters using:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%

During the rollout the following chain occurred:

New pod started and its /health returned 200.

Service/EndpointSlice began routing traffic to the new pod.

New pod had not finished cache warm‑up and DB connections were unstable.

Old pods were scaled down, instantly reducing available capacity.

Peak traffic hit the new pod, causing response‑time spikes, 502 errors, and OOM kills.

Rollback was triggered, but the old image had to be pulled again, taking >6 minutes.

The root causes were five engineering gaps: overly simple readiness probes, missing graceful shutdown, insufficient extra capacity, mis‑aligned HPA, and incomplete rollback of configuration, schema, and traffic policies.

Deployment controller fundamentals

Declarative reconciliation

Deployment is a declarative resource that defines desired replica count, pod template, and update strategy. The Deployment controller runs in kube-controller-manager and continuously reconciles current state to desired state via ReplicaSets.

Deployment
  ├── New ReplicaSet (new template)
  └── Old ReplicaSet (historical template)
        └── Pods

What triggers a new Revision

Only changes to the pod template (image, env vars, command, labels/annotations inside the template, sidecar config) create a new Revision. Changes to replica count, HPA scaling, or Deployment‑level metadata do not. Consequently kubectl rollout undo rolls back the pod template only; it does not revert HPA settings, DB schema, or external configuration.

Execution order of RollingUpdate

Create a new ReplicaSet.

Scale the new ReplicaSet up according to maxSurge.

Wait for new pods to become Available (Ready + minReadySeconds without crashes).

Scale down the old ReplicaSet according to maxUnavailable.

Repeat until the new ReplicaSet reaches the target replica count.

Note that Ready (probe passes) is not the same as Available (Ready for at least minReadySeconds and stable).

Parameter semantics

maxSurge specifies how many extra pods may be created during the rollout. Example: with 10 replicas and maxSurge: 30%, up to ceil(10 × 30%) = 3 extra pods can exist. Its purpose is to trade extra resources for smoother capacity transition, not to speed up the release.

maxUnavailable defines the maximum number of pods that may be unavailable during the rollout. Example: maxUnavailable: 20% with 10 replicas allows 2 pods to be down. It defines the acceptable capacity loss.

minReadySeconds forces a pod that is Ready to stay Ready for the given period before being considered Available. This protects against cold‑start latency, JIT warm‑up, cache miss spikes, etc.

progressDeadlineSeconds marks the Deployment as ProgressDeadlineExceeded if no progress is made within the time window. The controller continues retrying, but external systems (GitOps, CI/CD, alerts) must decide whether to pause or roll back.

Traffic switching is more than controller parameters

A pod termination affects multiple layers:

Pod readiness state changes.

EndpointSlice/Endpoints are updated. kube-proxy updates iptables/IPVS rules.

Ingress/Gateway/Service‑Mesh back‑end pools are refreshed.

Upstream client connection pools notice disconnections.

In‑flight requests on the old pod finish or are aborted.

Zero‑downtime therefore depends on correct readiness probes, proper preStop, sufficient terminationGracePeriodSeconds, and application‑level graceful shutdown logic.

Graceful shutdown sequence

Remove the instance from traffic entry points.

Stop accepting new requests.

Wait for in‑flight requests to finish.

Commit transactions, offsets, release locks and connections.

Exit the process.

Example implementation for a Spring Boot service:

@Component
public class TrafficDrainManager {
    private final AtomicBoolean shuttingDown = new AtomicBoolean(false);
    public boolean isShuttingDown() { return shuttingDown.get(); }
    @PreDestroy
    public void onShutdown() throws InterruptedException {
        shuttingDown.set(true);
        Thread.sleep(15000); // give upstream and kube‑proxy time to drain
        // additional steps: stop consuming messages, reject new tasks, wait for thread‑pool, flush logs
    }
}

@RestController
public class ReadinessController {
    private final TrafficDrainManager trafficDrainManager;
    public ReadinessController(TrafficDrainManager mgr) { this.trafficDrainManager = mgr; }
    @GetMapping("/health/readiness")
    public ResponseEntity<String> readiness() {
        if (trafficDrainManager.isShuttingDown()) {
            return ResponseEntity.status(503).body("draining");
        }
        // check DB pool, Redis, etc.
        return ResponseEntity.ok("ok");
    }
}

Key points: SIGTERM triggers readiness failure.

Wait for service‑discovery and connection‑drain.

Then let the process exit.

Readiness probe should verify real readiness

A probe that only returns 200 on /health provides little value. Production‑grade readiness must verify:

Web container startup.

Database connection pool health.

Redis/MQ/registry availability.

Local thread‑pool backlog.

Application is not in a "draining" state.

Readiness answers "Is it safe to send new traffic?" not "Is the process alive?"

Parameter sizing based on a capacity model

Core formula

Define: C: stable capacity per pod. Q: current total traffic. S: safety factor (1.2‑1.5). N: target stable replica count.

Requirement: N × C ≥ Q × S. If up to U pods may be unavailable, then (N‑U) × C ≥ Q × S, which yields the upper bound for maxUnavailable.

Example: QPS = 12 000, per‑pod capacity = 1 400, safety = 1.3 → required pods ≈ 11.14 → round up to 12. With 12 replicas, maxUnavailable must be 0; otherwise any pod loss breaches capacity.

Scenario‑specific recommendations

Core transaction chain

maxUnavailable: 0
maxSurge: 1
minReadySeconds: 30
progressDeadlineSeconds: 600

Typical stateless API

maxUnavailable: 10%
maxSurge: 20%
minReadySeconds: 15

Resource‑tight cluster

Reduce batch size.

Pre‑expand nodes.

Avoid maxSurge=0 and high maxUnavailable simultaneously.

A production‑ready Deployment template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
  labels:
    app: order-service
    tier: core-trade
  annotations:
    kubernetes.io/change-cause: "release-2026-06-05 optimize order write path"
spec:
  replicas: 12
  revisionHistoryLimit: 10
  minReadySeconds: 20
  progressDeadlineSeconds: 600
  selector:
    matchLabels:
      app: order-service
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: order-service
        version: v20260605
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: order-service
        image: registry.example.com/trade/order-service:v20260605
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "1000m"
            memory: "2Gi"
          limits:
            cpu: "2000m"
            memory: "4Gi"
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          failureThreshold: 30
          periodSeconds: 5
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
        env:
        - name: JAVA_OPTS
          value: "-XX:+UseG1GC -XX:MaxRAMPercentage=70"

Highlights: maxUnavailable: 0 guarantees no capacity loss. maxSurge: 1 adds only one extra pod at a time for easy observation. minReadySeconds: 20 prevents immediate traffic on a freshly Ready pod. startupProbe separates cold‑start failures from liveness. preStop + terminationGracePeriodSeconds give traffic‑drain and request‑cleanup time. revisionHistoryLimit retains enough versions for rollback without uncontrolled RS accumulation.

PodDisruptionBudget (PDB)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order-service-pdb
  namespace: production
spec:
  minAvailable: 11
  selector:
    matchLabels:
      app: order-service

PDB protects against node drain, cluster maintenance, or voluntary eviction by defining a system‑level minimum availability.

HPA interaction

During a rollout, HPA scaling can disturb the rollout cadence. Two common strategies:

Freeze HPA min/max limits during the release window.

Drive HPA with more stable metrics (QPS, queue depth, connection count) and use a smoothing window to avoid sudden scaling caused by a freshly started pod.

Four production capabilities the code must provide

Graceful refusal of new requests (readiness fails, gateway stops sending traffic, thread‑pool rejects new tasks, consumer pauses).

Waiting for in‑flight tasks to finish (long SQL/transactions, export jobs, streaming responses, batch message consumption, distributed lock sections).

Configuration compatibility first, then code compatibility; schema changes should be expand‑first then contract.

Clear separation of startupProbe, readinessProbe, and livenessProbe to avoid premature restarts, mis‑routing, and false health signals.

From manual commands to a release pipeline

Recommended release stages

Stage 1 – Pre‑release checks

Image built and vulnerability‑scanned.

Configuration compatibility verified.

Database changes are expand‑first.

Cluster has enough CPU/memory for maxSurge.

PDB, HPA, node taints/affinity satisfy constraints.

Stage 2 – Warm‑up & baseline scaling

Manually scale core services one extra replica.

Expand nodes if capacity is insufficient.

Pre‑pull images to reduce rollback pull risk.

Stage 3 – Batch rollout

Deploy 1 pod.

Observe 3‑5 minutes.

If stable, roll out another 10‑20%.

After stability, roll out the rest.

For finer‑grained canary, combine with Ingress/Nginx annotations, Service‑Mesh weight routing, Argo Rollouts, or Flagger.

Metrics to monitor during rollout

Business: success rate, order rate, TP99.

Application: JVM heap, thread‑pool queue length, DB connection pool, GC pauses.

Platform: pod Ready count, pod restarts, node CPU/memory, EndpointSlice change speed.

Release: updatedReplicas, readyReplicas, availableReplicas, unavailableReplicas, Deployment condition ProgressDeadlineExceeded.

Automatic stop‑loss

Common stop‑loss triggers (error‑rate spike > threshold within 5 min, TP99 > 2× baseline, pod restarts exceed threshold, readiness stays failing, consumer backlog surge). Actions: pause rollout, retain state, roll back to previous stable revision, lock the problematic version.

Rollback is more than kubectl rollout undo

Rollback can revert image, env vars, command args, and pod template, but cannot automatically revert database destructive changes, configuration‑center overrides, already sent messages, new data formats, or external traffic policies.

A full rollback involves four layers:

Image rollback (restore old pod template).

Configuration rollback (restore compatible config).

Data rollback / forward compensation (fix schema or erroneous data).

Traffic rollback (restore weight, circuit‑breaker, retry policies).

Retaining sufficient revisionHistoryLimit (10‑20 for core services, 5‑10 for ordinary services) is essential; setting it to 0 disables rollback capability.

Common pitfalls (top 10)

Readiness only checks HTTP 200, not critical dependencies.

No startupProbe, causing liveness to kill slow‑starting pods. preStop is a no‑op; application does not actually drain.

Too short terminationGracePeriodSeconds, process killed before cleanup. maxUnavailable set without capacity model, causing capacity dip.

Large maxSurge on a cluster without spare resources, leading to pending pods.

Rollback only image, ignoring config or data compatibility.

Concurrent rollout and HPA/Cluster‑Autoscaler scaling creates noisy metrics.

Ignoring resource consumption of terminating pods, eventually filling node memory.

No observability or automatic stop‑loss, allowing a single bad pod to affect the whole batch.

Production release checklist

Before release

Capacity assessment confirming maxUnavailable safety.

Node resources sufficient for maxSurge.

Startup, readiness, liveness probes in place.

Graceful shutdown implemented.

PDB defined.

Configuration and data forward‑compatible.

Rollback image and config versions retained.

During release

Batch‑by‑batch rollout with pause points.

Key metric dashboards active.

Automatic alerts and stop‑loss enabled.

During rollback

Determine whether only image or also config/data need rollback.

Use scripted, standard rollback procedures.

Ensure sufficient revisionHistoryLimit.

Validate that rollback image can be pulled and config is usable.

Conclusion

Kubernetes RollingUpdate solves the "replace replicas" layer, but a production‑grade release must also address capacity modeling, application‑level graceful shutdown, observability, and governance. When designed as a full‑stack system covering control plane, traffic layer, application lifecycle, and rollback governance, rolling updates become a reliable continuous‑delivery capability even under high‑concurrency workloads.