Evolving Jenkins Pipelines for Robust Deployments and Rolling Updates

Introduction

In the world of software deployment, Jenkins has become synonymous with automation. This blog, written by a senior operations engineer, explains how to evolve a Jenkins Pipeline so that it not only deploys applications to a running state but also performs health checks for Deployment and StatefulSet resources and configures rolling‑update strategies.

Initial Jenkins Pipeline Analysis

The original pipeline defines a basic CI/CD flow covering code checkout, Docker image build, push, and Kubernetes deployment. However, it lacks deployment‑status verification, which is crucial for ensuring stability.

pipeline {
    agent none // Use none at the top level, each stage will define its own agent.
    environment {
        REGISTRY = "swr.cn-north-4.myhuaweicloud.com/master-metaspace"
        KUBE_CONFIG = "--namespace=master-metaspace --context=master"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        BASE_WORKSPACE = "xxxxxxx"
    }
    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        // ... other stages omitted for brevity
    }
}

Evolution I: Introducing Probes for Deployment

Modern deployments require health checks after the Kubernetes Deployment reaches a READY state. A checkKubernetesResourceStatus method is added to poll the readyReplicas field via kubectl get. If the resource does not become ready within the timeout, the pipeline fails.

def checkKubernetesResourceStatus(String deploymentName, String namespace) {
    int attempts = 30
    int sleepTime = 10
    String readyReplicasJsonPath = ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        String statusCheck = sh(script: "kubectl get deployment ${deploymentName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"", returnStdout: true).trim()
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "Deployment ${deploymentName} is ready."
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}"
            sleep sleepTime
        }
    }
    error "Deployment ${deploymentName} did not become ready after ${attempts} attempts"
}

The stage Deploy game-ucenter now calls this method after deployment.

stage("Deploy game-ucenter") {
    when { environment name: 'game-ucenter', value: 'true' }
    agent { label "xxxx" }
    steps {
        deployToKubernetes("game-ucenter")
        checkKubernetesResourceStatus("game-ucenter", "master-metaspace")
    }
}

Evolution II: Compatibility with StatefulSet Health Checks

To support StatefulSet workloads, the status‑checking method is extended with a resourceType parameter, allowing the same logic to query either Deployment or StatefulSet resources.

def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30
    int sleepTime = 10
    String readyReplicasJsonPath = ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        String statusCheck = sh(script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"", returnStdout: true).trim()
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "${resourceType} ${resourceName} is ready."
            return
        } else {
            echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
            sleep sleepTime
        }
    }
    error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}

The Deploy datawriter-game-ucenter stage now uses the method with resourceType: "statefulset".

stage("Deploy datawriter-game-ucenter") {
    when { environment name: 'datawriter-game-ucenter', value: 'true' }
    agent { label "xxxxx" }
    steps {
        deployToKubernetes("datawriter-game-ucenter")
        checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
    }
}

Evolution III: Adding Rolling‑Update Strategy Checks

When updating a Deployment, a rolling‑update strategy minimizes downtime. A new method checkDeploymentUpdateStatus uses kubectl rollout status to monitor progress and fails the pipeline if the rollout does not succeed within the retry window.

def checkDeploymentUpdateStatus(String deploymentName, String namespace) {
    int attempts = 30
    int sleepTime = 10
    for (int i = 1; i <= attempts; i++) {
        String updateStatus = sh(script: "kubectl rollout status deployment/${deploymentName} --namespace=${namespace}", returnStdout: true).trim()
        if (updateStatus.contains("successfully rolled out")) {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}"
            sleep sleepTime
        }
    }
    error "Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts"
}

A unified checkRolloutStatus method is later introduced to handle both Deployments and StatefulSets.

def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30
    int sleepTime = 10
    if (!(resourceType in ["deployment", "statefulset"])) {
        error "Unknown resource type: ${resourceType}. Only 'deployment' and 'statefulset' are supported."
    }
    for (int i = 1; i <= attempts; i++) {
        String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
        String updateStatus = sh(script: rolloutCommand, returnStdout: true).trim()
        if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for ${resourceType} '${resourceName}' to successfully roll out. Attempt ${i}/${attempts}."
            sleep sleepTime
        }
    }
    error "${resourceType} '${resourceName}' did not successfully roll out after ${attempts} attempts in namespace '${namespace}'"
}

The final pipeline incorporates all these methods, providing a robust CI/CD flow that builds Docker images, deploys them, verifies resource readiness, and ensures successful rollouts for both Deployments and StatefulSets.

Summary

The blog demonstrates how a Jenkins Pipeline can evolve from a simple build‑and‑deploy script into a comprehensive, fault‑tolerant CI/CD solution. By adding health‑check functions, supporting multiple Kubernetes workload types, and integrating rolling‑update verification, the pipeline aligns with modern deployment practices and greatly improves reliability for operations teams.