Designing a Message Queue: Architecture, Protocols, Storage, and Consumption Management

Message queue (MQ) articles are selected and updated weekly, highlighting the importance of message queues as tools for system decoupling and traffic control, making them core components of distributed systems.

Understanding the implementation of a message queue is crucial, and this article examines the technology behind designing one.

Overall Design Idea

The design focuses on creating an end‑to‑end data flow where messages are produced, transmitted through a broker, and consumed.

The main components involved are the Producer (message producer), the Broker (message server), and the Consumer (message consumer).

Broker Design Focus

Key design points for the broker include message dumping (delivering messages at appropriate times), establishing a standard pattern to achieve decoupling, eventual consistency, and traffic shaping, and treating the broker as a message forwarder that performs two RPC calls (producer to broker, broker to consumer) with an optional third RPC for acknowledgment.

To realize the basic functions of a message queue—transmission, storage, and consumption—three design aspects must be addressed: communication protocol, storage selection, and consumption relationship maintenance.

Communication Protocol

Messages require a unified format, known as a message protocol, so that producers can construct and consumers can parse messages consistently.

Traditional protocols include XMPP and AMQP, while many modern queues implement custom protocols for performance.

JMS (Java Message Service) is an API standard that defines roles such as producers, consumers, and destinations, and supports two messaging models: point‑to‑point (queue) and publish‑subscribe (topic).

AMQP (Advanced Message Queuing Protocol) is a binary wire‑level protocol that standardizes message exchange across languages and systems; popular brokers like ActiveMQ and RabbitMQ support it.

JMS vs AMQP

JMS is limited to Java‑based platforms, whereas AMQP enables interoperability among diverse technologies.

Kafka Communication Protocol

Kafka uses a custom TCP‑based protocol designed specifically for its own requirements, handling producer, broker, and consumer interactions.

Storage Options

Possible storage backends for distributed systems include in‑memory, local file systems, distributed file systems, NoSQL stores, and relational databases.

In‑memory storage offers the highest speed but may tolerate message loss; databases provide reliable persistence suitable for high‑reliability scenarios such as transaction messages; HBase or Hadoop can be used for high‑throughput, less‑strict durability needs.

Consumption Relationship Management

Beyond basic message dumping, a queue must parse producer‑consumer relationships to deliver messages correctly, distinguishing between unicast (point‑to‑point) and broadcast (publish‑subscribe) models.

Maintaining subscription information often relies on external systems like Zookeeper, which notify consumers of changes in subscription state.

Advanced Features Required

Message ordering

Delivery reliability guarantees

Message persistence

Support for multiple messaging models

Multi‑instance clustering

Transactional capabilities

The article concludes with a summary of experiences and considerations for building a message queue from scratch.