How JD Cloud’s JCQ Redefines Cloud‑Native Messaging with Compute‑Storage Separation

Overview of JCQ

JCQ (JD Cloud Message Queue) is a cloud‑native distributed messaging middleware developed by JD Cloud. It is designed for high availability, data reliability, physical replica isolation, autonomous services, health reporting, minimal operations, container deployment, elastic scaling, tenant isolation, pay‑per‑use, and cloud account authorization.

Version History and Evolution

Development started in 2017 with version 1.0, released publicly in November 2018. Version 1.0 limited topics to a single server, unable to meet large‑scale Topic requirements.

Version 2.0, launched April 2019, added Topic scaling, load‑balancing of hot Topics across brokers, and hot broker traffic migration.

Version 3.0, released at the end of 2019, introduced compute‑storage separation, allowing independent deployment of compute (Broker) and storage (Store) components, improving upgrade flexibility and resource utilization.

Compute‑Storage Separation Benefits

In 2.0, compute and storage ran in the same process, so upgrading compute required a heavy storage restart involving data loading, index comparison, and dirty‑data truncation. 3.0 decouples them via RPC, so each can be upgraded independently without affecting the other.

Broker is a stateless service handling production, push, authentication, rate limiting, congestion control, and client load balancing. Store handles data write, replica sync, and read, and rarely needs changes after stabilization.

This separation enables mixed‑deployment with other products, sharing resource pools, and reduces hardware footprint by allowing independent scaling of compute and storage resources.

Scalability and High Availability

Multiple Brokers can cooperate to serve high‑fan‑out scenarios; each Broker handles a subset of SubTasks, reading from Store and pushing to consumers, achieving linear TPS growth across consumer groups.

Broker failover is lightweight and can complete within seconds. Physical anti‑affinity (cross‑rack, cross‑AZ) is considered during deployment, allowing flexible trade‑offs between availability and resource cost, such as M:1 cold‑standby.

Task Management and Load Balancing

Producer and consumer connections are abstracted as PubTask and SubTask, managed by a central Manager. The Manager monitors inbound/outbound traffic and bandwidth thresholds, migrating Tasks to less loaded Brokers when needed.

Raft‑Based Replication

JCQ 1.0 adopted the Raft algorithm for high availability and data consistency, using the Raft log as the message log. To overcome Raft’s sequential replication limits, JCQ expands the number of Raft groups (MultiRaft), achieving near‑linear concurrency growth.

Each StoreNode runs multiple Raft groups in parallel, sharing a NIO‑based communication thread pool, so adding groups does not linearly increase thread usage.

Cold‑Hot Data Separation

JCQ distinguishes hot data (frequently accessed) from cold data (rarely accessed). The Store Bridge abstraction allows plugging in various storage backends (Remote Raft Cluster, distributed file systems, S3) and offloading cold data to cheaper storage.

Future Directions

Planned evolutions include multi‑protocol compatibility, on‑demand auto‑scaling, deeper cloud‑native integration, exposing a JCQ Kernel with pluggable protocol adapters, and native Kubernetes Operator support for private and hybrid cloud deployments.