How to Build a Scalable CI/CD Pipeline for Hundreds of Daily Deployments on Kubernetes

Introduction

For developers, testers, and operations engineers familiar with CI/CD, this article explains how to design and implement a CI/CD solution in a Kubernetes environment that can handle hundreds of deployment requests each day.

Environment Background

Service runtime environment: Kubernetes

Number of application modules: 500+

Environments: Development, Test, Pre‑release, Production

Languages: Java, Go, Node.js (dynamic/static), PHP (multiple versions)

Functions covered: Code management, quality scanning, build, unit testing, image creation, gray/full release, rollback, permission control

Other requirements: Automatic CI trigger on code commit, manual trigger for other environments, release windows with emergency approval, progress bar, error output, independent multi‑environment deployment, flow control (gray then full), Feishu integration, cost efficiency

Tool Selection

Based on the extensive functional requirements, a mix of open‑source and commercial tools was chosen (the specific toolset is illustrated in the accompanying diagram).

Architecture Topology

The selected technology stack is combined into the architecture shown below:

To support the scenario of “high‑concurrency build tasks + efficient resource utilization”, CI jobs run in temporary pods inside Kubernetes; pods are automatically destroyed after the job finishes, maximizing the use of fragmented resources.

Standards and Conventions

Application naming: Unified names link CI job, Kubernetes service, deployment, container, and alert tags, eliminating extra mapping.

Environment and branch mapping: Each environment corresponds to a specific Git branch (illustrated in the diagram).

Language and compile conventions: Same language types share a unified build method; CI determines the compile command based on language.

Runtime environment: A common runtime definition includes working directory, start script (e.g., start.sh ${env}), port, log path, etc.

CI Implementation

Implementation follows the standard CI task flow.

1. Automated Build & Deployment

GitLab webhook links projects/environments to Jenkins jobs, automatically triggering builds on code push. Example configuration: gitlab configuration: (Diagram omitted for brevity) jenkins job configuration: (Diagram omitted for brevity)

2. Jenkins Dynamic Slave Creation

Using the Jenkins Kubernetes plugin and inbound feature, a temporary slave pod is created for each job and destroyed after completion. Required steps include creating RBAC for Jenkins in the cluster and defining a pod template per language.

# 1) Create k8s RBAC and Jenkins authentication
# 2) Define pod template for each language

3. CI Build Environment

Jenkins pipeline selects the appropriate language and environment, then runs the build steps. Example pipeline:

node() {
    stage('Set Agent') {
        AGENT_LABEL = "${params.LANGUAGE}"
    }
}

pipeline {
    agent {
        kubernetes {
            inheritFrom "${AGENT_LABEL}"
        }
    }
    stages {
        stage('Initialization') {
            steps {
                script {
                    if (params.BUILD_ENV == 'prod') {
                        env.GIT_BRANCH = 'release'
                    } else {
                        env.GIT_BRANCH = "${params.BUILD_ENV}"
                    }
                }
            }
        }
        stage('Git Clone Code') {
            steps {
                checkout([
                    $class: 'GitSCM',
                    branches: [[name: "${env.GIT_BRANCH}"]],
                    extensions: [
                        [$class: 'RelativeTargetDirectory', relativeTargetDir: 'code-clone'],
                        [$class: 'CloneOption', depth: 1, noTags: false, reference: '', shallow: true]
                    ],
                    userRemoteConfigs: [[credentialsId: 'jenkins', url: "${GITLAB_URL}"]]
                ])
            }
        }
        stage('Compile Package') {
            steps {
                script {
                    def LANGUAGE = "${params.LANGUAGE}"
                    container("${LANGUAGE}") {
                        sh "bash -x codecompile.sh ${LANGUAGE}"
                    }
                }
            }
        }
        stage('Image Build And Publish') {
            steps {
                script {
                    container('docker') {
                        sh "bash -x imagebuild.sh ${LANGUAGE}"
                    }
                }
            }
        }
    }
}

4. Quality Inspection & Unit Testing

SonarQube and JUnit are integrated via Jenkins plugins, with JaCoCo for coverage, forming a quality gate from unit testing to full code analysis.

Combining these tools reduces defect rates and improves maintainability and security.

5. Code Compilation and Image Building

Based on the standards, compile commands are generated per language:

function JUDGE_COMPILE_CMD() {
    if [[ ${LANGUAGE} == "java" ]]; then
        EXEC_COMPILE_CMD="mvn clean package -DskipTests"
    elif [[ ${LANGUAGE} == "golang1-17" || ${LANGUAGE} == "golang1-18" ]]; then
        EXEC_COMPILE_CMD="build.sh"
    elif [[ ${LANGUAGE} == "php" ]]; then
        EXEC_COMPILE_CMD="xxxxx.sh"
    elif [[ ${LANGUAGE} == "nodejs" ]]; then
        EXEC_COMPILE_CMD="npm install && npm i nodeinstall -g && nodeinstall"
    fi
}

Dockerfile generation and image build:

function DOCKER_FILE() {
    if [[ ! -f "deploy/Dockerfile" ]]; then
        echo "FROM ${BASE_IMAGE}" > Dockerfile
        echo "WORKDIR /home/services" >> Dockerfile
        echo "ADD start_env/ /home/services/" >> Dockerfile
        mkdir -p ./start_env/logs
        if [[ ${LANGUAGE} == "java" ]]; then
            cp -a code-clone/target/${SERVICE_NAME}.jar ./start_env/project/${SERVICE_NAME}.jar
        elif [[ ${LANGUAGE} == "nodejs" ]]; then
            cp -a code-clone/dist ./start_env/project
        else
            cp -a code-clone ./start_env/project
        fi
    else
        cp code-clone/deploy/Dockerfile ./
    fi
}

function IMAGE_BUILD() {
    docker build -t "${IMAGE_NAME}" .
    docker push ${IMAGE_NAME}
}

The code is provided for reference only.

6. Alternative CI Artifact Strategy

Instead of building images in CI, the pipeline can produce compiled packages, store them, and fetch them during release, depending on the environment.

CD Implementation

For continuous delivery, the cloud‑native tool ArgoCD is used instead of native Jenkins CD plugins.

1. CI/CD Integration

Separate Jenkins jobs for CI and CD; a successful CI triggers the CD job automatically.

build job: "${SERVICE_ENV}", parameters: [
    string(name: "IMAGE_VERSION", value: "${IMAGE_VERSION}")
]

2. Deployment

Version changes are committed to GitLab; ArgoCD detects the change and syncs the deployment. To avoid latency, a manual sync command is executed after committing.

argocd app sync "${JOB_BASE_NAME}" --grpc-web

3. Deployment Status & Error Capture

After triggering, deployment progress, status, and error logs are collected via ArgoCD API or CLI, and displayed in Jenkins.

4. Result Synchronization to Feishu

Build and release results are pushed to Feishu (or DingTalk, WeChat Work) via plugins; the article uses Feishu with a robot that can send messages to individuals.

5. Release Window and Permission Control

Release days (e.g., Tuesdays and Thursdays) are defined, and only responsible owners or team leads can trigger releases. Jenkins Role‑based Authorization Strategy plugin manages permissions; custom logic can enforce release windows.

Overall, the CI/CD solution can be adapted to various scenarios, and the author hopes the case study helps readers design their own pipelines.