Mastering Tencent Cloud Message Queue (CMQ): Architecture, Features & Use Cases

Background

Message queues are widely used for decoupling business logic, smoothing traffic spikes, flow control, and broadcasting messages, becoming a key internal communication method for many enterprises.

Common open‑source middleware includes RabbitMQ, Kafka, and RocketMQ, each with different strengths: Kafka emphasizes throughput but does not guarantee reliable storage and consistency, while RabbitMQ ensures reliable delivery at the cost of performance.

Tencent Cloud Message Queue (CMQ) is a high‑reliability, high‑availability, high‑performance distributed message‑queue service offering low coupling, strong consistency, scalability, and features such as push/pull consumption, message replay, delayed messages, publish‑subscribe, routing broadcast, and encryption.

Compared with Kafka, CMQ focuses on scenarios requiring high reliability (e.g., finance, transactions, orders); compared with RabbitMQ, CMQ provides significant improvements in availability and performance.

Underlying Architecture

CMQ’s architecture consists of sets, each containing three broker node replicas to ensure reliable storage and high availability. Data consistency is guaranteed by the Raft algorithm. A set follows the CP model of the CAP theorem; only when a majority of nodes are operational can messages be produced and consumed.

Practical Cases

1. Broadcast Pull Model

CMQ supports both queue (pull) and topic (push) models. The queue model is one‑to‑one pull, while the topic model is one‑to‑many push. CMQ combines them to create a broadcast‑pull model where a topic can have a queue subscriber, enabling pull‑based consumption even when push is not feasible.

# python sdk demo code: create subscription of queue protocol
my_sub = my_account.get_subscription(topic_name, subscription_name)
subscription_meta = SubscriptionMeta()
subscription_meta.Endpoint = "queue1"
subscription_meta.Protocal = "queue"
my_sub.create(subscription_meta)

2. Long Polling

In the queue model, consumers must poll for messages, which can waste resources if the queue is empty. CMQ’s long‑polling feature allows a consumer to block for up to a configured timeout (e.g., 10 seconds); if a message arrives during that period, it is returned immediately, otherwise the request returns after the timeout.

# python sdk demo code: receive message through long polling
pollingWaitSeconds = 3
recv_msg = my_queue.receive_message(pollingWaitSeconds)

3. Delayed Messages

CMQ can delay message visibility for a specified period after enqueueing, enabling simple implementation of scheduled tasks.

# python sdk demo code: send delayed message
msg_body = "I am delay message"
msg = Message(msg_body)
delaySeconds = 3
my_queue.send_message(msg, delaySeconds)

4. Message Replay

CMQ provides Kafka‑like replay capability, allowing consumption of messages that were previously deleted, based on a configurable backtrack timestamp.

# python sdk demo code: rewind the queue
# backtrack one hour
backTrackingTime = int(time.time()) - 3600
my_queue.rewindQueue(backTrackingTime)

5. Topic Routing Matching

CMQ’s topic model supports routing keys similar to RabbitMQ, where “*” matches a single word and “#” matches one or many words, enabling selective delivery based on patterns.

# python sdk demo code: set topic-subscription route-rule
my_sub = my_account.get_subscription(topic_name, subscription_name)
subscription_meta = SubscriptionMeta()
subscription_meta.Endpoint = "http://test.com"
subscription_meta.Protocal = "http"
subscription_meta.bindingKey = ['*.*.rabbit','lazy.#']
my_sub.create(subscription_meta)

message = Message()
message.msgBody = "route msg test"
my_topic.publish_message(message, 'quick.orange.rabbit')

6. Large Message Transmission

CMQ limits a single message to 1 MB. For messages larger than 64 KB, QPS is not guaranteed. Two solutions are recommended: (1) message fragmentation with header metadata for reassembly, and (2) storing the payload in Tencent Cloud Object Storage (COS) and sending the COS URL as the message.

7. Message Encryption

CMQ integrates with Tencent Cloud KMS. Two encryption approaches are available: client‑side encryption using a CMK to encrypt data before sending, and server‑side encryption where CMQ automatically encrypts/decrypts messages via KMS (still under development).