Boost System Performance: Mastering Message Queues and Their Use Cases

Highlights Overview

Message Queue Overview

Message Queue Use Cases

Message Middleware Examples

JMS Message Service (see second article)

Common Message Queues (see second article)

Reference Materials (see second article)

Summary of this Share (see second article)

1. Message Queue Overview

Message queue middleware is a crucial component in distributed systems, solving application coupling, asynchronous messaging, and traffic shaping, enabling high performance, high availability, scalability, and eventual consistency; it is indispensable for large‑scale distributed architectures.

Commonly used production message queues include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc.

2. Message Queue Use Cases

The following are typical scenarios where message queues are applied.

Asynchronous processing

Application decoupling

Traffic shaping

Message communication

2.1 Asynchronous Processing

Scenario: After user registration, a registration email and SMS need to be sent.

User registers, then needs email and SMS.

Traditional approaches:

1. Serial processing: write to DB, send email, then send SMS; response after all three complete.

2. Parallel processing: after DB write, send email and SMS concurrently; response after both complete, reducing time.

Assuming each step takes 50 ms, serial takes 150 ms, parallel about 100 ms. With a CPU capable of 100 requests per second, serial yields ~7 QPS, parallel ~10 QPS.

Summary: Traditional methods limit performance; introducing a message queue enables asynchronous handling, improving response time to the DB write latency (≈50 ms) and raising throughput to ~20 QPS.

2.2 Application Decoupling

Scenario: After an order is placed, the order system must notify the inventory system.

Traditional method: order system calls inventory API directly.

Problems: inventory unavailability causes order failure; tight coupling.

Solution with a message queue:

Order system persists the order, writes a message to the queue, and returns success to the user.

Inventory system subscribes to order messages, pulls or pushes updates, and adjusts stock accordingly.

If inventory is down, orders still succeed because the order system no longer depends on immediate inventory response, achieving decoupling.

2.3 Traffic Shaping

Used widely in flash‑sale or group‑buying events to handle sudden traffic spikes.

During a flash sale, massive traffic can crash the application; a queue buffers requests.

Control the number of participants.

Alleviate short‑term high traffic pressure.

Server writes incoming requests to the queue; if the queue is full, excess requests are dropped or redirected.

The sale service processes queued requests sequentially.

2.4 Log Processing

Message queues like Kafka are used for high‑volume log transmission.

Log collector client writes logs to Kafka.

Kafka stores and forwards log data.

Log processing application consumes logs from Kafka.

Example: Sina’s Kafka log processing pipeline includes Kafka, Logstash, Elasticsearch, and Kibana.

Kafka receives log messages.

Logstash parses logs into JSON for Elasticsearch.

Elasticsearch provides real‑time searchable storage.

Kibana visualizes the data.

2.5 Message Communication

Message queues provide efficient communication mechanisms, supporting point‑to‑point and publish‑subscribe patterns.

Point‑to‑point example:

Clients A and B share a queue for direct messaging.

Publish‑subscribe (chatroom) example:

Multiple clients subscribe to the same topic, enabling chatroom‑like communication.

3. Message Middleware Examples

3.1 E‑commerce System

The system uses highly available, persistent middleware such as ActiveMQ, RabbitMQ, or RocketMQ.

Core logic writes messages to the queue; optional confirmation ensures delivery.

Extended processes (SMS, delivery) subscribe to the queue and handle messages.

While decoupling introduces eventual consistency challenges, they can be addressed by combining queue processing with database writes.

3.2 Log Collection System

Components:

Zookeeper for service discovery and load balancing.

Log collector client pushes logs to Kafka.

Kafka cluster handles reception, routing, storage, and forwarding.

Storm (or other apps) consumes data from the queue.