Designing a Custom Message Queue: Principles and Practices

Message queues have become a core means of internal communication in enterprise IT systems, offering low coupling, reliable delivery, broadcasting, flow control, and eventual consistency, and are a primary method for asynchronous RPC.

Popular middleware includes ActiveMQ, RabbitMQ, Kafka, as well as Alibaba's Notify, MetaQ, and RocketMQ. This article does not enumerate all features of these products but instead explores the key considerations when designing a custom message queue, referencing many ideas from existing solutions.

It first discusses when a message queue is needed—typically for business decoupling, eventual consistency, broadcasting, and peak‑shaving. In scenarios requiring strong consistency and immediate results, RPC is more appropriate.

The article then focuses on the push model, analyzing design issues such as RPC, high availability, ordering, duplicate messages, reliable delivery, and consumer relationship parsing.

It also examines the pull model (e.g., Kafka) and its advantages, before moving on to advanced topics such as batch processing, flow‑control design, and fairness scheduling, which are deferred to a follow‑up article.

When to use a message queue

Use cases include business decoupling, eventual consistency, broadcasting, and peak‑shaving. If strong consistency and immediate response are required, RPC is preferable.

Decoupling

Decoupling allows a transaction to focus on its core flow while notifying dependent systems asynchronously, reducing the need for synchronous API calls.

Eventual consistency

Eventual consistency ensures that two systems end in the same state, even if intermediate steps are delayed. Systems like Notify and QMQ are built for high‑reliability notifications in transactional contexts.

Typical solutions involve either strong consistency via distributed transactions (costly) or eventual consistency using “record‑and‑compensate” patterns, where operations are logged and retried until successful.

Broadcast

Message queues enable broadcasting by allowing producers to publish once while multiple consumers subscribe independently, greatly reducing integration effort.

Peak‑shaving and flow control

When upstream and downstream processing capacities differ, a message queue acts as a funnel, buffering messages until the downstream can handle them, simplifying system design.

Designing a message queue

The core components are a broker and two RPC calls (send and acknowledge). High availability is achieved by making broker interfaces idempotent and leveraging shared storage (DB, distributed KV, or file system) for state persistence.

Storage options include in‑memory, distributed KV, disk, or relational databases, each with trade‑offs between performance, reliability, and cost.

Consumer relationship parsing involves supporting both unicast (point‑to‑point) and broadcast (one‑to‑many) models, typically using topics, partitions, and consumer groups as seen in JMS, Kafka, and RabbitMQ.

Advanced features

Reliable delivery is realized by persisting messages before sending and retrying until all endpoints confirm receipt. Duplicate handling can be addressed with version numbers or state‑machine logic.

Transaction support can be implemented via two‑phase commit (expensive) or local transactions with compensation, where message insertion and business updates share a single DB transaction.

Performance considerations

Async vs. sync RPC, client‑side thread pools, and server‑side asynchronous processing are discussed, highlighting how asynchronous models free threads and enable batch writes.

Batching strategies balance latency and throughput, using thresholds on message count, time, or arrival events, and may employ long‑polling to reduce pull‑model latency.

Push vs. Pull

Push models risk slow consumers causing broker backlog, while pull models give consumers control over consumption at the cost of increased latency. Hybrid approaches like long‑polling aim to combine the benefits of both.

Conclusion

The article provides a high‑level roadmap for designing a custom message queue, covering RPC, HA, storage, broadcast, acknowledgment, reliability, transaction handling, and performance optimization. A follow‑up will dive deeper into storage subsystem design, flow control, and fair scheduling.