A Systematic Overview of Message Queue Technical Points

Message queues serve as middleware for communication between services or applications, offering benefits such as decoupling, broadcast messaging, eventual consistency, and traffic shaping to overcome bottlenecks. This article does not dive into deep implementation details but instead presents a systematic outline of the key technical points, organized as a mind‑map.

1. Overall Architecture

Typical message‑queue systems consist of three roles: producer, broker, and consumer.

2. RPC Communication

For a detailed discussion, see the referenced article "Getting to Know RPC" (http://www.infoq.com/cn/articles/get-to-know-rpc).

3. Performance Guarantees

Performance focuses on latency and throughput. On the producer side, synchronous versus asynchronous sending and single‑message versus batch sending can be tuned; batch triggers may be buffer‑full or time‑window based. On the broker side, multiple topics, partitions/queues, and a divide‑and‑conquer strategy increase parallelism and reduce delivery pressure. Persistent messages can benefit from sequential I/O, page cache, and asynchronous flushing, though the latter risks data loss on power failure and may require high‑availability trade‑offs.

Consumer performance involves push or pull mechanisms, consumer groups for horizontal scaling, ordered or unordered consumption based on business needs, zero‑copy techniques to avoid user‑to‑kernel copies, batch pulling by consumers, and broker‑side filtering via tags or plugins.

4. High‑Availability Guarantees

Cluster HA focuses on the broker: a single master handles writes, with leader election either automatic (e.g., Kafka using ZooKeeper) or manual (e.g., RocketMQ using name servers). Data HA addresses persistent message storage, often using local sequential writes with master‑slave replication or multi‑copy strategies; synchronous multi‑write ensures no loss at the cost of performance, while asynchronous replication maximizes throughput but may lose data on master failure.

Cross‑datacenter HA requires minimizing MTTR and avoiding consumer blockage during failover.

5. Scalability Guarantees

Scalability is achieved through divide‑and‑conquer: multiple topics, partitions/queues, replicas, and slave or mirror nodes increase parallelism for both delivery and consumption.

6. Protocols

Producer, consumer, and broker communicate via protocols such as AMQP, STOMP, MQTT, HTTP, OpenWire (ActiveMQ), XMPP, or custom implementations. AMQP is comprehensive and used by RabbitMQ; Kafka and RocketMQ employ custom protocols.

7. Consumption Models

Three consumption patterns exist: point‑to‑point (P2P) FIFO queues, topic‑based publish/subscribe, and fan‑out broadcast.

8. Message Accumulation

Persistent messages stored on disk can use synchronous or asynchronous flushing; retention policies (time‑based or size‑based) clean up old data (e.g., Kafka, RocketMQ). Non‑persistent messages reside in memory and are removed after consumption.

9. Reliable Delivery

Producers may choose synchronous or asynchronous APIs, single or batch sends, and fire‑and‑forget modes. Error handling (e.g., connection failures, timeouts, publish failures) often involves retries, leading to possible duplicate deliveries.

Broker can guarantee zero data loss using synchronous multi‑write, multi‑replica strategies, and synchronous flushing, while also performing payload integrity checks such as CRC.

10. Reliable Consumption

Delivery semantics include at‑most‑once, at‑least‑once, and exactly‑once (the latter requiring coordinated streaming consumers and broker support, e.g., Storm Trident or Flink). Push versus pull modes affect latency and consumer load. Consumers track offsets, IDs, or timestamps for precise replay, acknowledge messages via ACK, and handle failures with NACK, requeue, or dead‑letter queues. Idempotent processing is required to tolerate duplicate consumption.

11. Message Types

Ordered messages can be partition‑ordered or globally ordered; unordered messages maximize throughput. Scheduled and transactional messages are also supported by systems like RocketMQ.

12. Message Query

RocketMQ provides lookup by message ID.

13. Ecosystem Integration

Rich client language support and tight integration with big‑data stacks (e.g., Kafka connectors for Spark, Storm, Flink, Kylin) illustrate the broader ecosystem.

14. Management Tools

Effective management includes topic, broker, cluster, permission/quota, multi‑tenant, client tools, monitoring, alerting, and console UI for comprehensive governance.

Source: http://neoremind.com/2018/03/消息队列技术点梳理/