How to Build a Highly Available RabbitMQ Cluster with HAProxy and Keepalived

1. Global Diagram

HAProxy is used for RabbitMQ load balancing and high availability, with Keepalived ensuring HAProxy's own HA.

Clients connect via a VIP; Keepalived routes the traffic to the active HAProxy instance, which balances requests across RabbitMQ nodes. If the Master Keepalived node or HAProxy fails, the Backup takes over and traffic is rerouted.

For higher reliability you can add multiple Backup Keepalived nodes for a one‑master‑many‑backup setup, though this increases hardware cost.

2. Install RabbitMQ

First install Erlang:

#rpm -Uvh https://mirrors.tuna.tsinghua.edu.cn/epel/epel-release-latest-7.noarch.rpm
#yum install erlang

Then download and install RabbitMQ:

#wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm
#yum install rabbitmq-server-3.6.6-1.el7.noarch.rpm
#service rabbitmq-server start
#service rabbitmq-server status

3. Configure RabbitMQ Cluster

Ensure each node can reach the others and share the same .erlang.cookie file.

cd /var/lib/rabbitmq
chmod u+w .erlang.cookie

Edit /etc/hosts on every node so the IPs map to the hostnames:

vim /etc/hosts

10.64.16.123    l-rabbitmq1
10.64.17.11     l-rabbitmq2

Join the second node to the cluster:

rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@l-rabbitmq1
rabbitmqctl start_app

Set one node to RAM mode for better performance:

rabbitmqctl change_cluster_node_type ram

4. Set Mirror Queue Policy

Define a policy so queues prefixed with ha. are mirrored on all nodes:

rabbitmqctl set_policy ha-all "^ha\." '{"ha-mode":"all","ha-sync-mode":"automatic"}'

5. Install and Configure Keepalived

Keepalived monitors server health and provides HA for HAProxy.

yum -y install keepalived
vim /etc/keepalived/keepalived.conf

Key parts of the configuration (MASTER and BACKUP instances) include:

global_defs {
    notification_email { [email protected] [email protected] [email protected] }
    notification_email_from [email protected]
    smtp_server 192.168.200.1
    smtp_connect_timeout 30
    router_id LVS_DEVEL
    vrrp_skip_check_adv_addr
    vrrp_garp_interval 0
    vrrp_gna_interval 0
}

vrrp_script chk_haproxy {
    script "service haproxy status"
    interval 1
    weight -2
}

vrrp_instance haproxy {
    state MASTER
    interface eth0
    virtual_router_id 108
    priority 100
    advert_int 1
    authentication { auth_type PASS auth_pass 1234 }
    track_script { chk_haproxy }
    virtual_ipaddress { 10.64.16.254 }
    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
}

# BACKUP instance is similar with state BACKUP and priority 99

The notify.sh script starts or stops HAProxy based on the node role.

6. Install and Configure HAProxy

Install HAProxy and add a configuration for stats, RabbitMQ admin UI, and the RabbitMQ cluster proxy.

yum install haproxy
vim /etc/haproxy/haproxy.cfg

Relevant snippet:

####################### HAProxy stats page #########################
listen http_front
    bind 0.0.0.0:1080
    stats refresh 30s
    stats uri /haproxy?stats
    stats realm "Haproxy Manager"
    stats auth admin:1234

##################### RabbitMQ admin UI #########################
listen rabbitmq_admin
    bind 10.64.16.254:15673
    server l-rabbitmq1 10.64.16.123:15672
    server l-rabbitmq2 10.64.17.11:15672

##################### RabbitMQ service proxy ####################
listen rabbitmq_cluster 10.64.16.254:5673
    mode tcp
    stats enable
    balance roundrobin
    option tcpka
    option tcplog
    timeout client 3h
    timeout server 3h
    timeout connect 3h
    server l-rabbitmq1 10.64.16.123:5672 check inter 5s rise 2 fall 3
    server l-rabbitmq2 10.64.17.11:5672 check inter 5s rise 2 fall 3

Restart HAProxy and verify the stats page at http://10.64.16.254:1080/haproxy?stats and the RabbitMQ admin UI at http://10.64.16.254:15673.

7. Test Producer and Consumer

Python producer using pika:

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('10.64.16.254',5673,virtual_host='/',credentials=pika.PlainCredentials(username='admin',password='1234')))
channel = connection.channel()
channel.queue_declare(queue='ha_1')
channel.basic_publish(exchange='',routing_key='ha_1',body='haha Hello World!')
connection.close()

Python consumer:

import pika, time

connection = pika.BlockingConnection(pika.ConnectionParameters('10.64.16.254',5673,virtual_host='/',credentials=pika.PlainCredentials(username='admin',password='1234')))
channel = connection.channel()
channel.queue_declare(queue='ha_1')

def callback(ch, method, properties, body):
    print("[x] Received %r" % body)
    time.sleep(15)
    ch.basic_ack(delivery_tag=method.delivery_tag)
    print("done")

channel.basic_consume(callback, queue='ha_1')
print("[*] Waiting for messages. To exit press CTRL+C")
channel.start_consuming()

Original source: http://dwz.date/bZGE, author: yuzhen0228