Understanding RabbitMQ Architecture: Exchanges, Queues, and Network Protocols

Preface

Message queues consist of five modules: communication protocol, network module, storage module, producer, and consumer. Producers and consumers must follow the protocol to communicate with the broker's network module for sending and receiving messages. The broker also stores and updates messages.

RabbitMQ System Architecture Overview

RabbitMQ is composed of three major modules: Producer, Broker, and Consumer.

Broker Module Overview

(1) The broker is the core of the messaging system, receiving, storing, and forwarding messages. It contains several key components:

Queue: stores messages, supporting in‑memory or persistent disk storage. The broker creates and deletes queues.

Exchange: the routing engine that forwards messages from producers to queues based on routing rules. It does not store data. The routing process is called Route, and setting routing rules is called Binding. Producers send messages to an exchange with a routing_key, and the exchange routes the message to one or more queues.

To illustrate Exchange, consider a postal sorting center as an analogy: the exchange is the sorting center, routing_key is the address on the mail, and queues are the delivery zones.

Roles and Components

Exchange: role – postal sorting center; function – receives mail from couriers and routes it according to the address (routing_key) to different delivery zones (queues).

routing_key: role – address information on the mail; function – specifies the destination.

Binding: role – connection between the sorting center and delivery zones; function – defines how mail is bound to a queue based on the routing_key.

Queue: role – storage point for delivery zones; function – stores mail awaiting pickup by couriers.

Specific Process

Mail reception: the producer (courier) sends mail to the exchange.

Mail sorting: the exchange routes the mail to the appropriate queue based on the routing_key.

Mail storage: the sorted mail is placed in the queue.

Mail delivery: the consumer (courier) retrieves mail from the queue and delivers it to the recipient.

The correspondence between the postal example and RabbitMQ concepts is:

Mail → Message

Sorting center → Exchange

Address → routing_key

Delivery zone → Queue

Binding → Binding

Exchange Types

Direct Exchange: routes messages with an exact routing_key match.

Fanout Exchange: broadcasts messages to all bound queues, ignoring routing_key.

Topic Exchange: routes messages based on pattern‑matching routing keys.

Headers Exchange: routes based on message header fields.

RabbitMQ Network Protocol

RabbitMQ data flow uses TCP (layer 4). It can also support protocols like STOMP, MQTT, and HTTP via plugins, but AMQP remains the default and primary protocol.

AMQP defines concepts such as Exchange, Exchange types, Binding, routing_key, and Queue.

RabbitMQ Network Module

The network layer distinguishes between Connection and Channel. A TCP connection (Connection) is established between a client and a broker. Within a connection, an AMQP channel (Channel) is created as a virtual connection for issuing commands.

Multiple connections are expensive; therefore RabbitMQ reuses a single TCP connection with multiple channels (NIO) to reduce overhead.

Tip: NIO (Non‑blocking I/O) consists of Channel, Buffer, and Selector. A single thread can monitor many channels.

RabbitMQ Storage

Storage consists of metadata storage and message data storage.

Metadata Storage

Each broker runs Mnesia, a distributed real‑time database that stores complete metadata. Synchronization issues can cause inconsistencies, leading to split‑brain or startup failures.

Message Data Storage

Messages are stored in queues. The persistent layer includes a queue index (rabbit_queue_index) and message store (rabbit_msg_store), which is split into persistent (msg_store_persistent) and transient (msg_store_transient) files. Data is appended to .rdp segment files, and indices are stored in .idx files.

RabbitMQ provides TTL for message expiration and a delayed‑delete mechanism that marks messages for removal and deletes whole segment files when all messages are marked. It also merges segment files when appropriate, balancing timely processing and storage efficiency for high‑throughput scenarios.

RabbitMQ Production and Consumption Mechanism

Producers and consumers interact directly with the broker without complex client addressing. The broker maintains full cluster metadata to route requests efficiently.

Producer Routing

Producers send data directly to the broker.

The server routes data to queues based on configured binding rules.

Routing logic resides on the server (Exchange and Route components), simplifying client implementation.

Consumer Modes

Push mode: broker pushes messages to consumers, rate controlled by basicQos.

Pull mode: consumers pull one message at a time.

Reliability Guarantees

Message acknowledgments (ACK) are supported.

Both automatic and manual ACK modes are available.

Consumers connect directly to queues without complex partitioning.