How to Build a Million‑Message‑Per‑Second RabbitMQ Service

Background

Leveraging RabbitMQ cluster horizontal scaling to balance traffic pressure can bring the cluster’s second‑level service capacity to millions of messages; Google and some production scenarios have performed similar experiments, and this article summarizes the experience and pitfalls.

RabbitMQ Overview

RabbitMQ is a message middleware based on the AMQP protocol, commonly used for storing and forwarding messages in distributed systems, offering ease of use, scalability, and high availability. It decouples service components so producers need not be aware of consumers.

Key concepts include Message, Queue, Binding, Exchange, Broker, Virtual‑host, Connection, and Channel.

Cluster Modes

In the default mode, a queue’s messages reside on a single node; consumers should connect to each node to avoid bottlenecks. Mirror (HA) queues replicate messages across multiple nodes, improving reliability at the cost of performance and network bandwidth.

How to Build a Million‑Message Service

Google’s experiment used 32 virtual machines (8‑core, 30 GB RAM) with 30 RAM nodes, 1 disc node, and 1 stats node, achieving over 1.3 M msgs/s production and consumption without memory pressure.

Key points for large clusters include distributing queues across nodes, using sharding plugins to split queues, and employing consistent‑hash exchanges for dynamic load balancing.

RabbitMQ Sharding Plugin

Enable the plugin (RabbitMQ ≥ 3.6.0) with: rabbitmq-plugins enable rabbitmq_sharding The plugin creates shard queues automatically when new nodes join the cluster, reducing single‑queue bottlenecks.

Define a policy to bind a custom exchange (e.g., shard.images) to create two shard queues per node:

set_policy images-sharding "^images" '{"sharding":2}'

Consistent‑Sharding Exchange

This exchange type distributes messages uniformly across queues using a hash of the routing key. Unlike the sharding plugin, queues must be created manually, but the exchange guarantees consistent placement of messages with the same routing key.

Reliability and Availability Discussion

Use publisher confirms and consumer acknowledgments to ensure message delivery. Enable confirm mode on channels with confirm.select and handle basic.ack, basic.nack, or basic.reject appropriately.

Configure heartbeat to detect broken TCP connections quickly; in AWS, ELB can use heartbeats to monitor RabbitMQ health.

Persist broker definitions and messages to disk to survive restarts, leveraging AMQP’s durability guarantees.

Scenario 1 – Confirm & Ack

Enable publisher confirms and consumer acks to guarantee that messages are persisted and processed.

Scenario 2 – Retry Mechanism

Use dead‑letter exchanges and TTL on retry queues to implement delayed retries without losing messages.

Scenario 3 – Delayed Tasks

Leverage TTL and dead‑letter routing to create delay queues that forward messages to the working queue after the timeout.

Scenario 4 – Cross‑Datacenter Federation

Enable the federation plugin to exchange messages between brokers without a cluster:

rabbitmq-plugins enable rabbitmq_federation

rabbitmq-plugins enable rabbitmq_federation_management

Scenario 5 – High‑Availability Queues

Configure mirrored queues (master‑slave) to survive node failures; adjust ha-promote-on-shutdown policy to prioritize either reliability or availability.

Performance tests show mirrored queues incur noticeable overhead; increasing prefetch can mitigate impact.

Performance Tips

Balance reliability and throughput by adjusting prefetch, adding nodes, or using sharding for high‑load scenarios.

Spring AMQP Integration

Spring AMQP provides AmqpTemplate and related APIs to interact with RabbitMQ, allowing seamless switching between AMQP brokers.