Kubernetes Node Maintenance: From Drain to True Zero‑Downtime Engineering

Problem with plain kubectl drain

In production clusters the simple three‑step sequence

kubectl cordon node-a
kubectl drain node-a --ignore-daemonsets --delete-emptydir-data
kubectl uncordon node-a

fails to guarantee zero disruption because:

New traffic may still be sent to a node that is being drained.

Pods receive SIGTERM but continue accepting requests.

PodDisruptionBudget (PDB), Deployment rollout, Cluster Autoscaler and HPA amplify scheduling and eviction behavior.

Long‑lived connections are cut mid‑flight.

Maintenance scripts delete Pods without verifying that business traffic has safely left.

Conclusion: drain is only one step in a larger shutdown chain that spans the control plane, data plane and application plane.

What kubectl drain actually does

Marks the node Unschedulable (cordon).

Enumerates evictable Pods on the node.

Calls the Eviction API (or falls back to delete) for each Pod.

Waits for Pods to terminate and be migrated.

The command guarantees that no new Pods are scheduled to the node and tries to move existing Pods, but it does not guarantee:

External traffic removal.

Application stop accepting new requests.

Connection draining.

Transaction completion.

Consumer pause.

Eviction vs direct kubectl delete pod

The Eviction API respects PodDisruptionBudget and performs a “voluntary interruption”. It considers replica lower bounds and fails or waits when the PDB disallows deletion. Production best practice is to prefer Eviction and avoid raw kubectl delete pod.

Zero‑interruption workflow

Zero‑interruption is achieved by first removing traffic from the business data plane, then letting Kubernetes perform the eviction.

New traffic must no longer be routed to the target node.

Existing connections are gracefully drained.

In‑flight requests, message consumption and transactions get a reasonable completion window.

Pod replicas always satisfy PDB and capacity constraints.

The maintenance workflow must be repeatable, observable and rollback‑able.

Six‑stage maintenance state machine

Pending : receive maintenance request; check change window and higher‑priority incidents.

PreCheckPassed : resource assessment (PDB, node redundancy, HPA min replicas, AZ distribution).

Cordoned : node frozen with kubectl cordon; verify no new Pods are scheduled.

TrafficDraining : data‑plane traffic offload; instance must stop receiving new traffic and pause consumers.

ReadyToEvict : verify in‑flight request count, connection count and lag are below thresholds.

Evicting : batch eviction; check whether eviction is blocked by PDB.

NodeMaintaining : host‑level actions (kernel patch, container‑runtime upgrade, image replace, disk expansion, reboot); monitor for timeouts.

Recovering : bring node back, wait for new Pods to become Ready, ensure metrics stabilize.

Completed : maintenance finished.

Failed : rollback or alert.

Only the drain step occupies one of these phases; the rest are engineering controls.

Three‑layer architecture

Control‑Plane : decides when to maintain, which node, whether eviction is allowed, and batch coordination.

Data‑Plane : responsible for pulling traffic off the target instances (service‑mesh, registration center, ingress).

Application‑Plane : implements graceful shutdown logic inside the Pod.

If any layer is missing, node maintenance can still cause service disruption.

Recommended component boundaries

Node Maintenance Orchestrator : state‑machine, batch control, rollback, audit (must).

Traffic Offloader : integrates with registration centers, service mesh, ingress to take instances offline (must).

Graceful Shutdown SDK : exposes a unified offline API, connection‑drain, readiness switch (strongly recommended).

Drain Executor : performs batched Eviction via the Eviction API (must).

Safety Guard : validates capacity, alerts, release‑window conflicts, node replaceability (must).

Observability Guard : monitors request zero‑rate, error‑rate, latency recovery (must).

Why not put all logic into preStop

preStop

duration is too short for complex orchestration.

Failure visibility is poor; not suitable for global audit.

Cannot coordinate multi‑system offload (e.g., Nacos + Mesh + consumer pause).

All logic runs after the Pod is already being terminated, leaving no chance to verify safe business exit before deletion.

Better approach: preStop only handles the final in‑process shutdown; traffic offload and business pause are triggered **before** the Pod is marked for eviction.

Graceful shutdown manager (Spring Boot example)

package com.example.cart.maintenance;

import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@Component
public class GracefulShutdownManager {
    private final AtomicBoolean draining = new AtomicBoolean(false);
    private final AtomicInteger inflightRequests = new AtomicInteger(0);
    private Set<String> bypassPaths = ConcurrentHashMap.newKeySet();

    public GracefulShutdownManager(MeterRegistry registry) {
        bypassPaths.add("/internal/maintenance/offline");
        bypassPaths.add("/actuator/health/liveness");
        Gauge.builder("app_inflight_requests", inflightRequests, AtomicInteger::get)
            .description("Current inflight requests")
            .register(registry);
        Gauge.builder("app_draining_state", draining, v -> v.get() ? 1 : 0)
            .description("1 means the instance is draining")
            .register(registry);
    }
    public boolean isDraining() { return draining.get(); }
    public int inflight() { return inflightRequests.get(); }
    public void enterDraining() { draining.set(true); }
    public boolean isBypassPath(String path) { return bypassPaths.contains(path); }
    public void incrementInflight() { inflightRequests.incrementAndGet(); }
    public void decrementInflight() { inflightRequests.decrementAndGet(); }
}

@Component
class GracefulShutdownFilter extends OncePerRequestFilter {
    private final GracefulShutdownManager manager;
    GracefulShutdownFilter(GracefulShutdownManager manager) { this.manager = manager; }
    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
            throws ServletException, IOException {
        String path = request.getRequestURI();
        if (manager.isDraining() && !manager.isBypassPath(path)) {
            response.setStatus(HttpStatus.SERVICE_UNAVAILABLE.value());
            response.getWriter().write("instance is draining");
            return;
        }
        manager.incrementInflight();
        try { filterChain.doFilter(request, response); }
        finally { manager.decrementInflight(); }
    }
}

@RestController
@RequestMapping("/internal/maintenance")
class MaintenanceController {
    private final GracefulShutdownManager manager;
    MaintenanceController(GracefulShutdownManager manager) { this.manager = manager; }
    @PostMapping("/offline")
    public String offline() {
        manager.enterDraining();
        return "accepted";
    }
}

The manager exposes two Prometheus metrics ( app_draining_state and app_inflight_requests) that the orchestrator can poll instead of using a blind sleep.

Readiness indicator reflecting draining state

package com.example.cart.maintenance;

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component("readinessState")
public class ReadinessStateHealthIndicator implements HealthIndicator {
    private final GracefulShutdownManager manager;
    public ReadinessStateHealthIndicator(GracefulShutdownManager manager) { this.manager = manager; }
    @Override
    public Health health() {
        if (manager.isDraining()) {
            return Health.outOfService().withDetail("state", "draining").build();
        }
        return Health.up().build();
    }
}

High‑concurrency pitfalls and mitigation

Batch eviction to avoid image‑pull, DB/Redis connection, JIT/class‑loading storms.

Health check after each batch.

Startup readiness gate to prevent traffic spikes.

Admission checks before maintenance

Is the system in a business peak?

Are critical alerts firing?

Any ongoing gray‑release, scaling or rollback?

Do the workloads on the node satisfy PDB?

Is there enough spare capacity for the displaced Pods?

Do stateful services need extra migration steps?

Typical guard rules (automated by the orchestrator):

Block maintenance if core error‑rate exceeds a threshold.

Block if ready node count falls below a safety threshold.

Allow only one node per AZ to be maintained simultaneously.

Limit the number of affected replicas per service.

Disallow overlap between release windows and maintenance windows.

Interaction with native Kubernetes controllers

PDB limits voluntary disruptions but does not guarantee traffic has been drained. Use minAvailable or maxUnavailable that matches real business redundancy and verify actual traffic off‑load before eviction.

Deployment : maxUnavailable together with PDB may slow down maintenance; the ready‑time of new replicas directly impacts drain efficiency.

StatefulSet : stable pod identity, storage dependencies, leader/follower roles may require explicit hand‑over, shard migration, and replica sync verification before eviction.

DaemonSet & static Pods : drain skips them by design because they are expected on every node. For critical daemonsets (logging, monitoring, CNI) decide whether they can be restarted together with the node and define a recovery order.

Special handling for long‑lived connections (gRPC, WebSocket, HTTP/2)

Because connections stay alive and multiplex many requests, simply removing a Pod from a Service does not stop traffic:

Existing connections continue to be used.

New requests may still be routed over the old connection.

Strategies:

Expose a graceful‑stop API on the gRPC server.

Switch WebSocket/SSE services to a “reject new sessions, keep old ones” mode.

Use mesh/proxy listener drain APIs.

Set a maximum lifetime for long connections during the maintenance window.

Key principle: a connection is part of the business entry; if it is not drained, the shutdown is not truly zero‑interruption.

Consumer / async job shutdown flow

Pause consumption.

Commit current offset or finish the current batch.

Wait for in‑flight message processing to finish.

Terminate the Pod.

Skipping these steps can cause duplicate consumption, half‑processed state and increased compensation pressure.

Observability – required metrics during maintenance

Node level : Ready status, CPU/Memory/Disk/Network, number of evicted Pods.

Instance level : in‑flight request count, QPS, 5xx ratio, P95/P99 latency, active connections, draining flag.

Cluster level : Pending Pods, scheduling failures, HPA fluctuations, image‑pull latency.

Async chain : consumer lag, retry queue backlog, dead‑letter count.

Suggested alerts:

If error rate rises within 3 min after maintenance start, abort further batches.

If Pod eviction wait exceeds a threshold, raise an alarm.

If the node stays cordoned but new Pods still land on it, alert.

If new replica Ready time spikes, alert.

Audit events should record who initiated maintenance, target node, timestamps, traffic off‑load time, drain time, eviction batch results, failure reasons and rollback actions. Emit these as Kubernetes Events, audit logs, change‑platform records and optionally ChatOps notifications.

Common anti‑patterns

Using sleep as the graceful exit mechanism – no business semantics, not observable, cannot adapt to different services.

Only configuring PDB without traffic off‑load – PDB protects replica count, not request success.

Direct kubectl delete pod instead of Eviction – bypasses PDB, uncontrolled risk.

One timeout for all services – cart, order, streaming have vastly different shutdown windows.

Orchestrator only watches Kubernetes, ignores business metrics – Pods may be deleted while users still see request failures.

Production checklist

Application integration

Expose a unified offline API or SDK.

Readiness must reflect the draining state.

Expose in‑flight request and connection count metrics.

Long‑connection services must support graceful stop.

Platform requirements

Node maintenance must be triggered via the orchestrator, not by manual delete pod.

Pre‑check capacity and PDB before starting.

All maintenance actions must be auditable.

Support batch, pause, resume and rollback.

Ops guidelines

Do not overlap maintenance windows with release windows.

Do not maintain multiple critical nodes in the same AZ concurrently.

After maintenance, perform business‑level acceptance, not just node Ready.

End‑to‑end maintenance flow example

Create a NodeMaintenanceJob CR.

Safety guard validates node health, PDB, capacity.

Cordon the node ( kubectl cordon).

For each workload call its offline hook (e.g., /internal/maintenance/offline).

Offloader (Nacos, Mesh, etc.) removes the instance from traffic.

Checker polls app_inflight_requests until below the target.

Batch Evictor removes Pods using the Eviction API.

Mark phase NodeMaintaining and perform host‑level actions (kernel patch, reboot).

After host work, uncordon the node and mark Completed.

If any step fails (off‑load error, drain timeout, eviction blocked, new replica not Ready, error‑rate spike) the controller moves the job to Failed, keeps the node cordoned, attempts to bring already‑drained instances back online, and raises an alert for manual intervention.

Tooling choices – from scripts to operators

Bash scripts : quick for small clusters, low frequency, but lack visibility, retry, rollback and audit.

CI/CD pipeline orchestration : good for teams with an existing change‑platform; provides approval and change record but still not a native runtime state machine.

Custom Controller / Operator : best for large clusters, multi‑service, multi‑protocol environments; offers declarative, observable, reusable maintenance as code. Higher initial cost but pays off at scale.

Script → platform‑level orchestrator → Operator is a typical migration path.

When zero‑interruption cannot be guaranteed

Business is not idempotent or lacks retry protection.

Long transactions exceed the acceptable maintenance window.

Stateful services lack role‑switch or data migration steps.

Cluster has no spare capacity.

Upstream clients keep long‑lived connections that cannot be forced to close.

Node failure is a sudden hardware fault, not a planned maintenance.

The realistic goal is to reduce the availability risk of planned node maintenance to within the business SLO that the system can tolerate.

Final takeaway

Turning node maintenance from a risky manual command into a production‑grade, zero‑interruption process requires a three‑layer engineering chain:

Control‑Plane : admission checks, batch strategy, PDB coordination, state‑machine.

Data‑Plane : traffic off‑load before eviction.

Application‑Plane : unified offline API, reject new requests, drain connections, expose exit metrics.

Platform Governance : audit, alerting, rollback, batch capability.

If you only see node maintenance as a “kubectl command”, it will always be high‑risk. When you combine a business exit protocol, a platform orchestrator and runtime observability, node maintenance becomes a safe, repeatable, and auditable operation.