How to Build a High‑Availability Jenkins Distributed Cluster with Kubernetes and LVS

Project Background

Jenkins, a Java‑based open‑source automation tool, is widely used in continuous integration. EOS, a code‑quality detection platform, uses Jenkins for remote task scheduling and scanning. As job volume, build frequency, and service pressure increased, resource consumption, configuration complexity, and maintenance costs became problematic.

Challenges

Using a single master for all jobs consumes excessive memory and CPU.

Master‑Slave distribution improves capacity but does not solve service distribution.

Adding more slave nodes introduces re‑configuration risks.

Different business lines require distinct node configurations, raising maintenance overhead.

Separate Jenkins instances per business line lead to resource duplication.

Containerization Implementation

To address these issues, a high‑availability Jenkins distributed cluster was designed. The overall architecture includes:

Master‑Slave architecture with label management: the master assigns jobs to specific slave nodes, improving master performance and build speed.

Kubernetes‑based containerization: Jenkins runs in containers, enabling elastic scaling and shared resources across the cluster.

LVS load balancing: a high‑performance, highly available server pool built on CentOS and LVS.

Master‑Slave Distributed Architecture

The Jenkins master triggers the build workflow: user initiates a scan, the server calls Jenkins, code is cloned, the scan script runs, and results are returned. As job count grows, Jenkins faces longer build times and instability due to external dependency fetching.

User (manual or automatic) triggers a scan on the server. Server invokes Jenkins service. Code is cloned. Scan script executes and analyses code. Scan results are returned to the server.

To improve, jobs are assigned to slaves via labels, and slaves are created dynamically based on labels.

Node Connection Methods

Jenkins provides two ways to connect slaves:

SSH launch : Master initiates a connection to the slave.

JNLP launch : Slave initiates a connection to the master using Java Web Start.

A custom automation script creates nodes and connects them using the JNLP method.

Kubernetes Containerization Practice

Kubernetes orchestrates Jenkins master and slave pods. The master runs on a designated node; when a build request arrives, the Kubernetes plugin creates a slave pod based on the job’s label, registers it with the master, and deletes the pod after the job completes, ensuring efficient resource usage.

Docker images are built for both master and slave:

Write a base Dockerfile (CentOS 7 with Jenkins dependencies).

Extend the base image to create master and slave images.

Build and push Docker images to a registry.

Write Kubernetes YAML files and set the correct image tags.

Deploy using the chosen orchestration tool.

LVS Load Balancing

LVS with Keepalived (IPVSADM) provides DR+weighted round‑robin load balancing, delivering high performance and high availability at the network layer. The primary LVS node supplies data sources (CFS) to all Jenkins servers, while the secondary node offers hot‑standby and remote backup via rsync + inotify. All Jenkins servers mount the CFS data source.

Project Results

Service high availability : If the Jenkins master fails, a new master container is automatically created.

Resource sharing : Jobs with the same label run on the same Docker host; multiple Docker images on a host support different job templates.

Efficient slave resource allocation : Slaves are created as Docker containers on demand and destroyed after use, eliminating long‑lived connections.

Easy slave scaling : When the Kubernetes cluster is resource‑constrained, additional nodes can be added to handle queued jobs.

Overall, the containerized solution delivers high availability, resource sharing, and scalable, automated Jenkins slave management.