How to Deploy a High‑Availability RabbitMQ Cluster on Kubernetes with NFS Storage

Introduction

RabbitMQ is an open‑source message broker that enables asynchronous communication between services, improving concurrency and decoupling micro‑services such as order and product services in e‑commerce.

RabbitMQ Features

Open source, high performance, fast speed, reliable delivery modes.

Rich API, seamless integration with Spring AMQP.

Cluster mode, HA mode, mirrored queues for data durability.

Typical Application Scenarios

Asynchronous processing – messages are stored in the broker and processed later.

Traffic shaping – during spikes (e.g., flash sales) the queue can reject excess requests to avoid overload.

Log processing – although Kafka is often preferred for logs.

Application decoupling – producers publish to the broker while multiple consumers subscribe independently, reducing coupling.

Cluster Authentication

Nodes authenticate using an Erlang cookie, a shared secret stored in /var/lib/rabbitmq/.erlang.cookie (or /root/.erlang.cookie). All nodes must have identical cookie content.

Cluster Modes

Standalone mode.

Standard cluster (no high availability).

Mirrored cluster (high availability, most common).

Deployment Overview

For Kubernetes the recommended deployment is a StatefulSet with a Headless Service, using NFS for persistent storage.

1. Version Note

Older RabbitMQ versions (< 3.6.x) used the autocluster plugin, which is no longer maintained. Use the peer‑discovery subsystem for 3.7.x and newer.

2. Deployment Methods

Four methods exist (IP, DNS, StatefulSet + Headless Service, hostname). This guide uses the StatefulSet + Headless Service approach.

3. NFS Persistent Storage

# mkdir -p /data/storage/k8s/rabbitmq

Create an RBAC object for the NFS provisioner (nfs‑rbac.yaml) and apply it: # kubectl apply -f nfs-rbac.yaml Create a StorageClass that uses the NFS provisioner:

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: rabbitmq-nfs-storage
provisioner: rabbitmq/nfs
reclaimPolicy: Retain

# kubectl apply -f rabbitmq-nfs-class.yaml

4. NFS Client Provisioner

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-nfs-client-provisioner
  namespace: wiseco
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq-nfs-client-provisioner
  template:
    metadata:
      labels:
        app: rabbitmq-nfs-client-provisioner
    spec:
      serviceAccount: nfs-provisioner
      containers:
      - name: rabbitmq-nfs-client-provisioner
        image: registry.cn-hangzhou.aliyuncs.com/open-ali/nfs-client-provisioner
        env:
        - name: PROVISIONER_NAME
          value: rabbitmq/nfs
        - name: NFS_SERVER
          value: 172.16.60.238
        - name: NFS_PATH
          value: /data/storage/k8s/rabbitmq
        volumeMounts:
        - name: nfs-client-root
          mountPath: /persistentvolumes
      volumes:
      - name: nfs-client-root
        nfs:
          server: 172.16.60.238
          path: /data/storage/k8s/rabbitmq

# kubectl apply -f rabbitmq-nfs.yml

5. RabbitMQ StatefulSet

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-management
  namespace: wiseco
spec:
  ports:
  - port: 15672
    name: http
  selector:
    app: rabbitmq
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: wiseco
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 4369
    name: epmd
  - port: 25672
    name: rabbitmq-dist
  clusterIP: None
  selector:
    app: rabbitmq
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: wiseco
spec:
  serviceName: "rabbitmq"
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: "app"
                operator: In
                values: [rabbitmq]
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: rabbitmq
        image: rabbitmq:3.7-rc-management
        env:
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: RABBITMQ_ERLANG_COOKIE
          value: "YZSDHWMFSMKEMBDHSGGZ"
        - name: RABBITMQ_NODENAME
          value: "rabbit@$(MY_POD_NAME)"
        ports:
        - name: http
          containerPort: 15672
        - name: amqp
          containerPort: 5672
        livenessProbe:
          tcpSocket:
            port: amqp
          initialDelaySeconds: 5
        readinessProbe:
          tcpSocket:
            port: amqp
          initialDelaySeconds: 15
        volumeMounts:
        - name: rabbitmq-data
          mountPath: /var/lib/rabbitmq
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-data
    spec:
      accessModes: [ReadWriteMany]
      resources:
        requests:
          storage: 10Gi
      storageClassName: rabbitmq-nfs-storage

# kubectl apply -f rabbitmq.yml

6. Verification

Check pods and services:

# kubectl get pods -n wiseco -o wide | grep rabbitmq
# kubectl get svc -n wiseco | grep rabbitmq

Use rabbitmqctl cluster_status inside each pod to confirm that all three nodes are part of the cluster.

Access the management UI at http://<node_ip>:32513 (NodePort 15672) with user guest and password guest.

7. Management Commands

Users:

# rabbitmqctl add_user Username Password
# rabbitmqctl delete_user Username
# rabbitmqctl change_password Username NewPassword
# rabbitmqctl list_users

Roles (tags) – administrator, monitoring, policymaker, management, etc. Set with: # rabbitmqctl set_user_tags User Tag1 Tag2 Permissions – configure, write, read on exchanges/queues:

# rabbitmqctl set_permissions -p VHostPath User ConfP WriteP ReadP
# rabbitmqctl list_permissions [-p VHostPath]
# rabbitmqctl list_user_permissions User
# rabbitmqctl clear_permissions [-p VHostPath] User

Node type (disk vs RAM):

# rabbitmqctl stop_app
# rabbitmqctl change_cluster_node_type dist   # or ram
# rabbitmqctl start_app

8. Fault Simulation

Delete a pod (e.g., rabbitmq-0) to simulate node failure, observe cluster status via rabbitmqctl cluster_status and the UI. The UI shows red (failed), yellow (recovering), and green (healthy) states.

9. Client Access

Clients can connect to any node (e.g., rabbitmq-0.rabbitmq.wiseco.svc.cluster.local:5672). For transparent failover, expose a single address via a load balancer such as LVS or HAProxy that proxies the three node ports.