Migrating Jenkins Slave from Virtual Machines to Kubernetes for Efficient CI/CD

The company discovered low utilization of AWS VMs used as Jenkins slaves and decided to migrate the slaves to a Kubernetes container cloud to reduce costs, improve resource usage, and enable elastic scaling for increasing test job demand.

Overall Architecture : Jenkins follows a master‑slave model where each slave is now a Kubernetes pod. The master runs on a VM, while slaves are created on demand as pods using the Kubernetes plugin. Pods contain three containers – a JNLP container for Jenkins communication, a Python‑36 container for Python tests, and a Java‑8 container for Java tests.

Jenkins Master Configuration : Install the Kubernetes plugin via the Manage Plugins page, then configure the cloud connection with a name, Kubernetes API server URL, and a base64‑encoded certificate. The credentials are generated by extracting

# echo certificate-authority-data的内容 | base64 -D > ~/ca.crt

and creating a PFX file with

# openssl pkcs12 -export -out ~/cert.pfx -inkey ~/client.key -in ~/client.crt -certfile ~/ca.crt

. After adding the certificate credential, set the Jenkins URL and the maximum number of concurrent pods, then test the connection.

Pod Template (YAML) defines the pod’s namespace, three containers, shared NFS volume, resource limits, and node selectors. Example snippet:

apiVersion: v1
kind: Pod
metadata:
  namespace: sqe-test
spec:
  containers:
  - name: jnlp
    image: swc-harbor.nioint.com/sqe/jnlp-slave:root_user
    volumeMounts:
    - mountPath: /home/jenkins/agent
      name: jenkins-slave
  - name: python36
    image: swc-harbor.nioint.com/sqe/automation_python36:v1
    command:
    - cat
    tty: true
    env:
    - name: WORKON_HOME
      value: /home/jenkins/agent/.local/share/virtualenvs/
    volumeMounts:
    - mountPath: /home/jenkins/agent
      name: jenkins-slave
    resources:
      limits:
        cpu: 300m
        memory: 500Mi
  - name: java8
    image: swc-harbor.nioint.com/sqe/automation_java8:v2
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: /home/jenkins/agent
      name: jenkins-slave
  volumes:
  - name: jenkins-slave
    nfs:
      path: /data/jenkins-slave-nfs/
      server: 10.125.234.64
  nodeSelector:
    node-app: normal
    node-dept: sqe

Custom Docker Images :

JNLP image (root user):

FROM jenkinsci/jnlp-slave:latest
LABEL maintainer="[email protected]"
USER root

Python‑36 image with pipenv:

FROM python:3.6.4
LABEL maintainer="[email protected]"
USER root
RUN pip --upgrade install pip
RUN pip3 install pipenv

Java‑8 Maven image with timezone and locale settings and custom settings.xml:

FROM maven:3.6.3-jdk-8
LABEL maintainer="[email protected]"
USER root
RUN mv /etc/localtime /etc/localtime.bak && \
    ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone
RUN apt-get update && apt-get install locales -y && \
    echo "zh_CN.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
ADD settings.xml /root/.m2/
COPY jacoco-plugin/jacococli.jar /usr/bin
RUN chmod +x /usr/bin/jacococli.jar

Jenkins Pipeline (Jenkinsfile) uses the Kubernetes agent block to request a pod:

pipeline {
    agent {
        kubernetes {
            cloud 'kubernetes-bj'
            label 'SEQ-AUTOTEST-PYTHON36'
            defaultContainer 'python36'
            idleMinutes 10
            yamlFile "jenkins/jenkins_pod_template.yaml"
        }
    }
    environment {
        git_url = '[email protected]:liuchunming033/seq_jenkins_template.git'
        git_key = 'c8615bc3-c995-40ed-92ba-d5b66'
        git_branch = 'master'
        email_list = '[email protected]'
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '30'))
        timeout(time: 30, unit: 'MINUTES')
        disableConcurrentBuilds()
    }
    stages {
        stage('拉取测试代码') {
            steps { git branch: "${git_branch}", credentialsId: "${git_key}", url: "${git_url}" }
        }
        stage('安装测试依赖') { steps { sh "pipenv install" } }
        stage('执行测试用例') { steps { sh "pipenv run py.test" } }
    }
    post { always { container("jnlp") { allure includeProperties: false, jdk: '', report: 'jenkins-allure-report', results: [[path: 'allure-results']] } } }
}

Performance Optimizations focus on keeping idle pods (idleMinutes) and using the same label to reuse pods, as well as tuning Jenkins master load‑calculation parameters to accelerate pod provisioning. Example JVM options added to the Jenkins start command:

-Dhudson.model.LoadStatistics.clock=2000 \
-Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000 \
-Dhudson.slaves.NodeProvisioner.initialDelay=0 \
-Dhudson.model.LoadStatistics.decay=0.5 \
-Dhudson.slaves.NodeProvisioner.MARGIN=50 \
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

After completing the configuration, creating a pipeline job and triggering a build will automatically provision a Kubernetes pod as a Jenkins slave, run the test stages, generate reports, and clean up the pod when idle, achieving higher resource efficiency and elastic scaling compared to static VM slaves.

The article concludes that using Kubernetes for CI/CD not only solves the inefficiencies of VM‑based Jenkins slaves but also provides a flexible foundation for large‑scale performance testing environments.