How TDMQ RocketMQ Lite Topic Keeps Millions of Channels Lightweight (Part 2)

Recap of the previous article

In the first part we examined two real AI‑native scenarios—multi‑agent collaboration and large‑model memory writing—and saw that Lite Topic plays a crucial role in the communication chain.

Lite Topic overview

To accommodate long‑running, multi‑round interactions, Apache RocketMQ extends its Light Message Queue (LMQ) capabilities with a new Lite Topic feature. The design introduces a two‑level hierarchy: a Topic (parent) provides routing, business isolation, and aggregated metrics, while each Lite Topic serves as the actual message storage carrier. This separation gives a global view at the Topic level and fine‑grained monitoring at the Lite Topic level.

Puzzle 1 – How millions of Lite Topics stay "light"

Automatic lifecycle : When a client sends a message, the broker checks whether the Lite Topic exists and creates it on‑the‑fly if needed. A TTL (time‑to‑live) can be set at creation; if the latest message in a Lite Topic is older than the TTL, the topic is considered expired and is removed asynchronously, along with its consumption offsets, indexes, and in‑memory subscription state. This prevents unused topics from lingering.

Storage reuse : All messages for Lite Topics are appended to the same physical queue used by LMQ, preserving global ordering. The existing LMQ storage layer is reused, so Lite Topic inherits the proven, million‑level queue storage model without adding new structural complexity. Failover during broker restarts or single‑node failures automatically redirects writes to other healthy nodes.

Puzzle 2 – How massive subscriptions avoid overloading the broker

Client‑level subscription management : Unlike the traditional Consumer Group model, Lite Topic pushes subscription state down to the client identifier (ClientId). This eliminates the need for the broker to maintain group‑wide consistency, dramatically reducing management overhead at million‑scale.

Ready Event Set (event‑driven merge pull) : When a new message is written and its consumption index is built, the broker immediately dispatches a "ready event" to the relevant subscribers. Each subscriber maintains a Ready Event Set that records ready events for all its Lite Topics. On a POP request, the broker reads directly from this set, merging pulls across multiple Lite Topics. The benefits are twofold: (1) a single request can fetch messages from many topics, cutting RPC calls; (2) empty Lite Topics generate no pull traffic, avoiding useless storage queries.

Puzzle 3 – Exclusive consumption and the "latest‑connection‑wins" rule

Traditional RocketMQ uses a shared consumption mode where all clients in a Consumer Group balance the load. For session‑preserving use cases, this fragments a single conversation across multiple SSE gateways, making it impossible to reconstruct the full dialogue.

To solve this, Lite Topic introduces an Exclusive consumption mode : only one client under a Consumer Group may consume messages from a given Lite Topic. When the broker detects multiple clients, it selects the most recent connection as the exclusive consumer and instructs older clients to cancel their subscriptions. This "latest‑connection‑wins" rule enables seamless "break‑resume" behavior without any business‑side coordination code—clients simply call seek(lastReceivedOffset+1) and the protocol handles the rest.

Key differences from traditional Topic

The added Lite Topic capabilities bring three main advantages over classic Topics: (1) two‑level data isolation while preserving the original storage model; (2) single‑client exclusive subscriptions for fine‑grained session control; (3) automatic lifecycle management that creates topics on send and deletes them after TTL expiry.

Design principle and future outlook

All Lite Topic features map to observable, evolvable mechanisms inside the RocketMQ kernel, reflecting a design principle of keeping product‑level capabilities transparent to the core engine. As AI technologies evolve, RocketMQ’s Lite Topic is expected to support LLM streaming, agent state management, RAG data foundations, model inference pipelines, and multimodal data processing, providing a solid infrastructure foundation for AI‑native applications.

用户断线 → 网关 A 仍持有订阅
用户重连命中网关 B → 网关 B 发起排他订阅
   → Broker 触发 "最新连接胜出"
   → 自动踢出网关 A
   → 网关 B 接管该会话的后续 token