Configuring Jenkins High Availability with HAProxy and NFS

The article introduces the need for high availability (HA) in Jenkins CI/CD pipelines, describing the risk of a single point of failure and proposing a simple HA solution using two Jenkins master servers with one acting as active and the other as passive.

Background : Continuous Integration (CI) accelerates development, and Jenkins is a popular CI tool. To avoid downtime, the author suggests configuring two Jenkins masters with HAProxy for load balancing.

What is High Availability? HA means a system can run continuously without failure. Jenkins downtime impacts DevOps, so HA is required. The article then explains how HAProxy works, routing client requests based on server health.

Jenkins HA Configuration : The setup consists of three layers – HAProxy as the entry point, two Jenkins masters sharing the same JENKINS_HOME via NFS, and an AWS EFS (or NFS) mount point accessible to both masters.

Detailed Technical Implementation

1. Host Information

192.168.1.100  HA
192.168.1.102  Jenkins01
192.168.1.103  Jenkins02

2. Deploy NFS Service (on 192.168.1.103)

# Install NFS utilities
yum -y install nfs-utils

# Create shared directory
mkdir /data
echo "/data 192.168.1.0/24(rw)" >/etc/exports
exportfs -r
exportfs -v

# Start and enable NFS service
systemctl start nfs-server
systemctl enable nfs-server

# Create mount point and mount
mkdir /var/lib/data
mount -t nfs 192.168.1.103:/data /var/lib/data

3. Deploy Jenkins Service (on 192.168.1.102)

# Install JDK
# Download Jenkins package
https://mirrors.tuna.tsinghua.edu.cn/jenkins/

# Set JENKINS_HOME
export JENKINS_HOME=/var/lib/data/jenkins

# Start Jenkins
nohup java -jar jenkins-2.190.2.war -Djenkins.slaves.DefaultJnlpSlaveReceiver.disableStrictVerification=true --httpPort=8080 &

4. Deploy HAProxy

# Compile and install HAProxy
tar zxf haproxy-2.0.8.tar.gz
cd haproxy-2.0.8
make TARGET=linux-glibc PREFIX=/usr/local/haproxy
make install PREFIX=/usr/local/haproxy
cp ./examples/haproxy.init /etc/init.d/haproxy
chmod 755 /etc/init.d/haproxy
useradd -r haproxy
mkdir /etc/haproxy

# Generate /etc/haproxy/haproxy.cfg
global
    maxconn 20000
    ulimit-n 16384
    log 127.0.0.1 local0
    uid 200
    gid 200
    chroot /var/empty
    nbproc 4
    daemon

listen admin_stats
    bind *:8001
    mode http
    option httplog
    stats enable
    stats refresh 5s
    stats uri /haproxy?stats
    stats auth admin:123456
    stats realm welcome\ Haproxy
    stats admin if TRUE

listen jenkins_agent
    mode tcp
    bind *:8181
    balance roundrobin
    server jenkins01 192.168.1.102:8181 weight 1 maxconn 10000 check inter 1s
    server jenkins02 192.168.1.103:8181 weight 1 maxconn 10000 check inter 1s

frontend jenkins_proxy
    bind *:80
    log global
    option redispatch
    option forwardfor
    option http-server-close
    maxconn 8000
    timeout client 30s
    default_backend jenkins_proxy
    reqadd X-Forwarded-Proto:\ http

backend jenkins_proxy
    balance roundrobin
    option httpchk GET /login
    server jenkins1 192.168.1.102:8080 check inter 1000 rise 1 fall 1 weight 30
    server jenkins2 192.168.1.103:8080 check inter 1000 rise 1 fall 1 weight 30

# Start HAProxy
systemctl start haproxy

# Configure rsyslog for HAProxy logs
# /etc/rsyslog.d/haproxy.conf
$ModLoad imudp
$UDPServerRun 514
$UDPServerAddress 0.0.0.0
local0.* /var/log/haproxy.log
systemctl restart rsyslog
systemctl haproxy

The HAProxy admin interface is accessible at http://192.168.1.100:8001/haproxy?stats .

Test Results : If Jenkins01 fails, HAProxy redirects traffic to Jenkins02, but a manual reload on Jenkins02 is required to synchronize data. HAProxy HTTP proxy provides master health checks, while TCP proxy handles automatic agent connections.