Automate Multi‑Instance Kubernetes Deployments with a Single Jenkins Pipeline Template

Introduction

In continuous integration and deployment we often need to roll out many similar components to a Kubernetes cluster. By using a Jenkins pipeline and a single .tpl template, we can automate this process without creating separate template files for each component.

Problem Background

Multiple game‑center services (e.g., game-ucenter-*) share almost identical Kubernetes YAML files, differing only in identifiers such as instance numbers. Maintaining a separate template for each instance is inefficient and error‑prone.

Solution Overview

Leverage sed and loop structures in a Jenkins pipeline to generate many configuration files from one template and deploy each service instance automatically.

Step 1: Define Jenkins Pipeline

The pipeline defines environment variables and two helper functions: buildAndPushDockerImage and deployToKubernetes. These functions build Docker images and apply Kubernetes manifests.

pipeline {
    agent none // each stage defines its own agent
    environment {
        REGISTRY = "xxxx/master-metaspace"
        KUBE_CONFIG = "--namespace=master-metaspace --context=master"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        BASE_WORKSPACE = "xxxxxxx"
    }
    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader', value: 'true' }
                    steps { buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE) }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter', value: 'true' }
                    steps { buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE) }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps { buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE) }
                }
            }
        }
        stage('Development Deployment') {
            parallel {
                stage('Deploy datawriter-game-ucenter') {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label "huaweiyun-xx" }
                    steps { deployToKubernetes("datawriter-game-ucenter") }
                }
                stage('Deploy dataloader-game-ucenter') {
                    when { environment name: 'dataloader', value: 'true' }
                    agent { label "huaweiyun-xx" }
                    steps { deployToKubernetes("dataloader-game-ucenter") }
                }
                stage('Deploy game-ucenter') {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("game-ucenter-1")
                        deployToKubernetes("game-ucenter-2")
                        // ... additional instances as needed
                    }
                }
            }
        }
    }
}

Helper Functions

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: $(pwd)'"
    sh "cd ${workspacePath}/${imageName} && docker build --build-arg NODE_ENV=${imageName} -t ${REGISTRY}/${imageName}:${tag} ."
    withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u ${dockerUser} -p ${dockerPassword} ${REGISTRY}"
        sh "docker push ${REGISTRY}/${imageName}:${tag}"
    }
}

def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/${data}/g' ${templateFile} > ${outputFile}"
    sh "sudo kubectl apply -f ${outputFile} ${KUBE_CONFIG}"
}

Step 2: Extend deployToKubernetes for Multiple Instances

def deployToKubernetes(String kubernetesComponent, int instance = 1, int totalInstances = 1) {
    if (instance < 1) { error("Instance number must be greater than 0") }
    String nameSuffix = totalInstances > 1 ? "-${instance}" : ""
    String outputFileName = "${kubernetesComponent}${nameSuffix}.yaml"
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${outputFileName}"
    String nameReplacement = "${kubernetesComponent}${nameSuffix}"
    sh "cat ${templateFile} |
        sed 's/{data}/${data}/g' |
        sed 's/name: ${kubernetesComponent}/name: ${nameReplacement}/g' |
        sed 's/app: ${kubernetesComponent}/app: ${nameReplacement}/g' > ${outputFile}"
    sh "kubectl apply -f ${outputFile} ${KUBE_CONFIG}"
}

Step 3: Prepare the Template File

The game-ucenter.tpl contains generic Deployment and Service definitions with placeholders such as game-ucenter-1 that will be replaced by the script.

# Example snippet of game-ucenter.tpl
apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-ucenter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-ucenter
  template:
    metadata:
      labels:
        app: game-ucenter
    spec:
      containers:
      - name: game-ucenter
        image: xxxx/xxx/game-ucenter:{data}
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "4096M"
            cpu: "2000m"
          limits:
            memory: "4096M"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /test.html
            port: 80
          initialDelaySeconds: 20
          periodSeconds: 120
        readinessProbe:
          httpGet:
            path: /test.html
            port: 80
          initialDelaySeconds: 20
          periodSeconds: 120
---
apiVersion: v1
kind: Service
metadata:
  name: game-ucenter
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: game-ucenter

Step 4: Execute the Pipeline

When the pipeline reaches the "Development Deployment" stage, it loops to create and apply game-ucenter-1.yaml through game-ucenter-20.yaml, deploying the corresponding number of service instances.

By using a single template and Jenkins automation, the need to maintain separate files for each instance is eliminated, improving efficiency and reducing the risk of mistakes.

Note

The code and commands are illustrative; adapt them to your specific Jenkins environment and Kubernetes cluster. Test thoroughly before deploying to production.