How Tencent’s TDMQ Achieves Cloud‑Native, High‑Performance Messaging for Finance

Financial‑grade Cloud‑Native Messaging Requirements

TDMQ is designed for cloud‑native scenarios that demand multi‑language, multi‑protocol compatibility, unlimited topics, and unlimited consumers, with rapid on‑demand scaling to satisfy the core requirements of modern message queues.

Background: Tencent Billing Platform

Tencent Billing supports billions of daily transactions across 180+ countries, handling over a trillion business codes and more than one million compute tenants. It offers four versions (basic, enterprise, overseas, industry) and processes transactions up to hundreds of millions per request.

Message‑Queue Use Cases in Tencent Billing

Online services: asynchronous processing, application decoupling, traffic shaping, and message communication for end‑to‑end payment flows.

Offline near‑real‑time services: multi‑party reconciliation, mixed‑payment settlement, and real‑time monitoring of transaction logs.

Other scenarios: data synchronization across multiple DB sources and log aggregation.

TDMQ Functional Architecture

TDMQ provides the same core capabilities as mainstream queues while adding features required for financial workloads:

High consistency & reliability : supports delayed messages, ordered messages, and tenant‑level rate‑limiting.

Dual consumption models : Queue (unordered, shared) and Stream (strict order, exclusive) models.

Multi‑protocol access : native SDKs for Java, C++, Go, Python, Node.js, plus Kafka 2.0+ and RabbitMQ 4.0+ client compatibility.

Management console : message content query, traceability, and open management APIs.

Storage‑Compute Separation Architecture

Built on Apache Pulsar, TDMQ separates brokers (stateless routing and control) from bookies (persistent storage). Brokers store only metadata in ZooKeeper and route client requests to the appropriate broker for each topic. Bookies store messages in immutable ledgers, enabling efficient data migration, horizontal scaling, and independent scaling of compute and storage resources.

Consistency Model

TDMQ uses a quorum‑based write protocol similar to Raft. When writing to a topic, three parameters are specified:

Ensemble Size (E) : number of storage nodes involved.

Write Quorum Size (W) : number of nodes that must receive the write.

Ack Quorum Size (A) : number of nodes that must acknowledge the write.

The broker acts as the Raft leader for a topic, ensuring ordered, id‑based message IDs (ledger‑id + entry‑id) and preventing split‑brain scenarios via a fence mechanism on bookies.

Cross‑Region Disaster Recovery

TDMQ supports both same‑city active‑active and cross‑city active‑active deployments. Data replicas can be placed in multiple regions (e.g., two replicas in Shenzhen, one in Shanghai) and the ack strategy can be configured to require acknowledgments from both local and remote replicas, achieving true city‑level disaster tolerance.

Read‑Only Broker Design for Massive Consumers

To support tens of thousands of consumers without degrading write performance, TDMQ introduces read‑only brokers that synchronize topic metadata from the write broker and read from closed ledgers. Real‑time consumption of the open ledger is achieved by periodically syncing the latest confirmed entry ID (lastAddConfirmedID) from the write broker to bookies.

Database Synchronization Practice (TDSQL)

TDSQL uses TDMQ as an incremental log layer for cross‑city replication. A dedicated binlogproducer process on the standby node parses MySQL binlogs and pushes them to TDMQ. The consumer process ( binlogconsumer) pulls the logs in the target city and replays them, guaranteeing at‑least‑once delivery, ordered replay, and idempotent handling of conflicts.

Production reliability: no binlog loss on MySQL node failure; no out‑of‑order messages.

Consumption reliability: hash‑based sharding of rows ensures parallel replay while preserving per‑row order; idempotent logic deletes conflicting rows before re‑inserting.

FAQ (Selected Technical Q&A)

Is TDMQ used for all Tencent Cloud billing? Yes, it underpins both online transaction processing and offline reconciliation.

Can TDMQ synchronize databases? It supports multi‑source DB sync across regions, leveraging its cross‑city deployment and high‑concurrency hash routing.

How does TDMQ differ from Kafka? Kafka excels at raw throughput, while TDMQ adds financial‑grade reliability, strong consistency, and native multi‑protocol support.

How does TDMQ guarantee data consistency? Through the quorum‑based write model, ordered message IDs, and acknowledgment of the previous confirmed ID on each write.

Does TDMQ support delayed consumption? Yes, it provides delayed messages for retry scenarios and time‑based consumption such as flash‑sale events.

Overall, TDMQ demonstrates how a storage‑separated, cloud‑native message queue can satisfy the stringent reliability, consistency, scalability, and multi‑region disaster‑recovery needs of large‑scale financial systems.