Master HAProxy 3.0: From System Tuning to Advanced Load‑Balancing Practices

Overview

HAProxy has evolved from a simple TCP/HTTP proxy (v1.4) to a full‑featured load balancer that supports QUIC/HTTP‑3, multithreading, and a rich Runtime API in version 3.0. It is optimized for pure proxy scenarios where fine‑grained traffic control and raw throughput are critical.

Key Technical Features of HAProxy 3.0

Native QUIC/HTTP‑3 – experimental but stable after three months of production testing.

Mature multithreaded architecture – nbthread can be set to the number of CPU cores (max 64) without lock‑contention issues.

Runtime API enhancements – add, remove or drain back‑end servers without restarting.

Improved memory pools – stable memory usage for long‑lived connections.

Typical Use Cases

API gateways that need rate limiting, circuit breaking, and per‑service routing.

Micro‑service entry points with dynamic service‑discovery integration.

Database proxies for MySQL/PostgreSQL read‑write splitting and connection pooling.

WebSocket long‑connection handling.

High‑performance HTTP proxying where raw throughput matters.

HAProxy is not ideal for complex URL rewrites or static file serving; those tasks are better handled by Nginx.

Environment Requirements

HAProxy 3.0.5 (stable release, Jan 2025)

Rocky Linux 9.3 or Ubuntu 24.04 LTS (Rocky Linux recommended)

Linux kernel 5.14+ with io_uring support

CPU 4‑core minimum (8‑core recommended for production)

Memory 4 GB+ (adjust according to maxconn)

GCC 12+ for source compilation

OpenSSL 3.0+ for QUIC support

Step‑by‑Step Deployment

System Optimisation

Before installing HAProxy, tune kernel and sysctl parameters to avoid bottlenecks under high load.

# /etc/sysctl.d/99-haproxy.conf
fs.file-max = 2097152
fs.nr_open = 2097152
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 65535
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fastopen = 3
net.netfilter.nf_conntrack_max = 1048576

Apply the settings: sysctl -p /etc/sysctl.d/99-haproxy.conf Increase file‑descriptor limits for the haproxy user:

# /etc/security/limits.d/99-haproxy.conf
haproxy soft nofile 1048576
haproxy hard nofile 1048576
haproxy soft nproc 65535
haproxy hard nproc 65535

Installing HAProxy 3.0

Rocky Linux 9 ships an older HAProxy package, so use the official COPR repository or compile from source.

Method 1 – Official COPR repository (recommended)

# Add the repository
dnf install -y epel-release
dnf copr enable -y bors/haproxy
# Install HAProxy 3.0
dnf install -y haproxy30

Method 2 – Compile from source (maximum performance)

# Install build dependencies
DNF install -y gcc make pcre2-devel openssl-devel systemd-devel lua-devel readline-devel zlib-devel
# Download source
cd /usr/local/src
wget https://www.haproxy.org/download/3.0/src/haproxy-3.0.5.tar.gz
tar xzf haproxy-3.0.5.tar.gz
cd haproxy-3.0.5
# Build with required features
make -j$(nproc) TARGET=linux-glibc \
    USE_OPENSSL=1 USE_PCRE2=1 USE_SYSTEMD=1 \
    USE_LUA=1 USE_QUIC=1 USE_PROMEX=1 \
    USE_THREAD=1
make install PREFIX=/usr/local/haproxy

Create a systemd service unit ( /etc/systemd/system/haproxy.service).

[Unit]
Description=HAProxy Load Balancer
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/haproxy
ExecStartPre=/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/local/haproxy/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
LimitNOFILE=1048576

[Install]
WantedBy=multi-user.target

Core Configuration

HAProxy configuration is divided into four sections: global, defaults, frontend, and backend. The following example shows a production‑grade setup.

Global Settings

# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0 info
    log /dev/log local1 notice
    user haproxy
    group haproxy
    daemon
    nbthread 8            # set to number of CPU cores (max 64)
    cpu-map auto:1/1-8 0-7
    maxconn 100000        # adjust according to memory (≈32 KB per conn)
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    tune.ssl.default-dh-param 2048
    tune.bufsize 32768
    tune.maxrewrite 1024
    tune.http.maxhdr 128
    lua-load /etc/haproxy/lua/init.lua

Defaults

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor
    option redispatch
    retries 3
    timeout connect 5s
    timeout client 60s
    timeout server 60s
    timeout http-request 10s
    timeout http-keep-alive 30s
    timeout queue 30s
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

Frontend Example (HTTP & HTTPS)

# HTTP frontend – port 80
frontend http_front
    bind *:80
    acl is_api path_beg /api/
    acl is_static path_end .css .js .png .jpg .gif .ico
    acl is_websocket hdr(Upgrade) -i WebSocket
    acl host_app1 hdr(host) -i app1.example.com
    acl host_app2 hdr(host) -i app2.example.com
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    acl is_abuse sc_http_req_rate(0) gt 100
    http-request deny deny_status 429 if is_abuse
    http-request track-sc0 src
    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Real-IP %[src]
    use_backend api_servers if is_api
    use_backend static_servers if is_static
    use_backend ws_servers if is_websocket
    use_backend app1_servers if host_app1
    use_backend app2_servers if host_app2
    default_backend web_servers

# HTTPS frontend – port 443
frontend https_front
    bind *:443 ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    http-request set-header X-Forwarded-Proto https
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
    use_backend api_servers if is_api
    default_backend web_servers

Backend Examples

# Web servers – round‑robin
backend web_servers
    balance roundrobin
    cookie SERVERID insert indirect nocache
    option httpchk GET /health HTTP/1.1
Host:\ localhost
    http-check expect status 200
    server web1 192.168.1.11:8080 weight 100 cookie web1 check inter 3s fall 3 rise 2
    server web2 192.168.1.12:8080 weight 100 cookie web2 check inter 3s fall 3 rise 2
    server web3 192.168.1.13:8080 weight 100 cookie web3 check inter 3s fall 3 rise 2
    server web_backup 192.168.1.20:8080 backup

# API servers – least‑connections
backend api_servers
    balance leastconn
    option httpchk GET /api/health HTTP/1.1
Host:\ localhost
    http-check expect status 200
    http-reuse aggressive
    retry-on conn-failure empty-response response-timeout 502 503 504
    server api1 192.168.1.21:3000 check inter 2s fall 2 rise 2 maxconn 1000
    server api2 192.168.1.22:3000 check inter 2s fall 2 rise 2 maxconn 1000

# WebSocket backend – source‑hash for session affinity
backend ws_servers
    balance source
    timeout tunnel 3600s
    timeout server 3600s
    server ws1 192.168.1.31:8080 check
    server ws2 192.168.1.32:8080 check

# Static assets – gzip compression and cache headers
backend static_servers
    balance roundrobin
    compression algo gzip
    compression type text/css text/javascript application/javascript
    http-response set-header Cache-Control "public, max-age=86400"
    server static1 192.168.1.41:80 check
    server static2 192.168.1.42:80 check

Monitoring and Statistics

A dedicated stats frontend can be secured with IP whitelisting and basic authentication.

# Stats page – port 8404
frontend stats
    bind *:8404
    mode http
    acl allowed_ip src 10.0.0.0/8 192.168.0.0/16
    http-request deny unless allowed_ip
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if TRUE
    stats auth admin:your_secure_password
    http-request use-service prometheus-exporter if { path /metrics }

Best Practices & Caveats

Health‑check optimisation : use HTTP checks that hit real business endpoints; deeper checks can validate downstream dependencies.

Connection reuse : http-reuse safe for conservative reuse, aggressive only when back‑ends fully support keep‑alive.

Graceful drain : set a server to drain via Runtime API, wait for active sessions to finish, then switch to maint.

Log format : define a JSON log format for easy ingestion into ELK/Loki.

Troubleshooting Common Errors

502 Bad Gateway – usually a backend timeout; increase timeout server or optimise the backend.

503 Service Unavailable – all backends down; verify health‑check configuration.

Connection exhaustion – maxconn too low; raise the limit and ensure kernel buffers are sufficient.

Cookie loss – mis‑configured cookie attributes; add httponly secure flags as needed.

SSL handshake failures – incomplete certificate chain; include intermediate certificates.

CPU pinned to a single core – multithreading not enabled; set nbthread appropriately.

Runtime API commands for live inspection:

# Show statistics
 echo "show stat" | socat stdio /run/haproxy/admin.sock
# List active sessions
 echo "show sess" | socat stdio /run/haproxy/admin.sock
# Show server state
 echo "show servers state" | socat stdio /run/haproxy/admin.sock

Backup & Recovery

#!/bin/bash
BACKUP_DIR="/backup/haproxy"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p ${BACKUP_DIR}
# Backup configuration files
tar czf ${BACKUP_DIR}/haproxy_config_${DATE}.tar.gz \
    /etc/haproxy/ \
    /etc/sysctl.d/99-haproxy.conf \
    /etc/security/limits.d/99-haproxy.conf
# Backup certificates
tar czf ${BACKUP_DIR}/haproxy_certs_${DATE}.tar.gz /etc/haproxy/certs/
# Keep last 30 days
find ${BACKUP_DIR} -type f -mtime +30 -delete

Recovery steps: reinstall the same HAProxy version, extract the archives, validate the configuration with haproxy -c -f /etc/haproxy/haproxy.cfg, then start the service.

Conclusion

HAProxy 3.0 provides a robust, multithreaded, and feature‑rich platform for high‑performance load balancing. By following the system tuning, installation, configuration, monitoring, and backup guidelines above, operators can achieve reliable, scalable, and secure traffic management for API gateways, micro‑services, databases, and web applications.