How to Build a High‑Availability HAProxy + Varnish + Nginx Stack with Keepalived

Implementation Points

(1) Separate static and dynamic DiscuzX traffic using load balancing; (2) Test adding Varnish cache between HAProxy and backend hosts; (3) Provide a topology design; (4) HAProxy configuration requirements: enable stats, custom 403/502/503 error pages, choose appropriate backend scheduling, log properly, and use keepalived for HA.

Topology Diagram

Component Overview

Two keepalived master nodes provide high availability for two HAProxy instances, each exposing a virtual IP (VIP) – 10.1.253.11 and 10.1.253.12. HAProxy handles incoming requests, performs static‑dynamic separation, forwards image requests to a Varnish cache, and routes dynamic traffic to two HTTPD hosts.

Varnish caches responses from backend Nginx servers and balances between them while performing health checks.

Nginx serves cached static images and provides an NFS‑mounted directory for the DiscuzX attachment folder.

Web servers (websrv) run Apache, MySQL, PHP, host the DiscuzX application, and serve unseparated static resources such as CSS.

NFS Server Configuration

Install NFS service yum install nfs-utils Configure NFS export

/etc/exports
/data/discuz 10.1.253.66(rw,no_root_squash) 10.1.253.67(rw,no_root_squash)

Create the Apache user and grant permissions:

useradd -s -u 48 -g 48 apache
setfacl -m u:apache:rwx /data/discuz

Start NFS service

systemctl start nfs.service

Web Server (websrv) Setup

Install AMP stack and DiscuzX

yum install httpd mysql php php-mysql php-xcache
mysql -uroot -p -e "CREATE DATABASE ultrax;GRANT ALL ON ultrax.* TO [email protected].%.% IDENTIFIED BY 'ultraxpass';FLUSH PRIVILEGES"

Mount NFS to the attachment directory (details omitted for brevity).

Nginx Configuration

Install Nginx and configure a virtual host that points its document root to the NFS‑shared directory.

URL rewriting can be performed in HAProxy, Varnish, or Nginx; the guide recommends doing it in HAProxy or Varnish to simplify management.

Sample virtual host

server {
    listen 82;
    server_name localhost;
    location / {
        root   /data/discuz;
        index  index.html index.htm;
    }
    location ~* \.(jpg|jpeg|gif|png)$ {
        root /data/discuz;
        rewrite ^/.*forum/(.*)$ /$1 break;
    }
}

Test by replacing the host part of the URL with the Nginx IP and accessing the image directly.

Varnish Cache Server

Install Varnish (after enabling the EPEL repository) and configure runtime parameters:

/etc/varnish/varnish.params
VARNISH_LISTEN_PORT=80
VARNISH_STORAGE="malloc,128M"

Configure backend servers and URL rewriting in /etc/varnish/default.vcl:

vcl 4.0;
import directors;
backend nginx1 { .host = "10.1.253.29"; .port = "81"; .probe = ok; }
backend nginx2 { .host = "10.1.253.29"; .port = "82"; .probe = ok; }
sub vcl_init { new RR = directors.round_robin(); RR.add_backend(nginx1); RR.add_backend(nginx2); }
sub vcl_recv {
    set req.backend_hint = RR.backend();
    if (req.url ~ "(?i)\.(jpg|jpeg|gif|png)$") {
        set req.url = regsub(req.url, "/.*attachment/(.*)", "/$1");
    }
}

Start Varnish and verify that requests are served from the cache.

HAProxy Configuration

Install HAProxy: yum install haproxy Key parts of /etc/haproxy/haproxy.cfg:

frontend main *:80
    acl url_static_beg path_beg -i /data/attachment
    acl url_static_end path_end -i .jpg .gif .png .css .js
    use_backend static if url_static_beg url_static_end
    default_backend dynamic
    errorloc 503 http://10.1.253.29:82/errorpage/503sorry.html
    errorloc 403 http://10.1.253.29:82/errorpage/403sorry.html
    errorloc 502 http://10.1.253.29:82/errorpage/502sorry.html

backend dynamic
    balance roundrobin
    server web1 10.1.253.66:81 check cookie amp1
    server web2 10.1.253.66:82 check cookie amp2

backend static
    balance roundrobin
    server ngx1 10.1.253.29:81 check
    server ngx2 10.1.253.29:82 check

listen stats
    bind *:9000
    stats enable
    stats uri /admin?stats
    stats refresh 10s
    stats admin if TRUE
    stats hide-version

Enable HAProxy stats page for monitoring.

Keepalived High‑Availability for HAProxy

Install keepalived: yum install keepalived Configure a dual‑master keepalived setup in /etc/keepalived/keepalived.conf (excerpt):

global_defs {
    notification_email { root@localhost }
    notification_email_from [email protected]
    smtp_server localhost
    smtp_connect_timeout 30
    router_id node1
    vrrp_mcast_group4 224.22.29.1
}

vrrp_script chk_down {
    script "[[ -f /etc/keepalived/down ]]&& exit 1 || exit 0"
    interval 1
    weight -5
}

vrrp_script chk_haproxy {
    script "killall -0 haproxy && exit 0 || exit 1"
    interval 1
    weight -5
}

vrrp_instance VI_1 {
    state MASTER
    interface eno16777736
    virtual_router_id 10
    priority 96
    advert_int 10
    authentication { auth_type PASS; auth_pass 1a7b2ce6; }
    virtual_ipaddress { 10.1.253.11 dev eno16777736; }
    track_script { chk_down; chk_haproxy; }
}

vrrp_instance VI_2 {
    state BACKUP
    interface eno16777736
    virtual_router_id 11
    priority 100
    advert_int 11
    authentication { ... }
    virtual_ipaddress { 10.1.253.12 dev eno16777736; }
}

Start keepalived and test failover by creating/removing a down file in /etc/keepalived/. Logs show VIP migration between HAProxy nodes.

Testing Results

Both VIP1 and VIP2 successfully serve the DiscuzX site through HAProxy. Image resources are served either by Varnish or Nginx depending on cache state.

Conclusion

HAProxy is a high‑performance reverse proxy that can handle HTTP and TCP traffic, provide health checks, and expose a convenient stats page. URL rewriting can be performed in HAProxy, Varnish, or Nginx; placing the rewrite logic in HAProxy or Varnish simplifies management of many backend hosts. The single‑process, event‑driven architecture of HAProxy enables it to handle massive concurrent connections with fine‑grained backend scheduling.