How QBus Redefines Messaging: A Deep Dive into 360’s Custom Kafka‑Based Queue

As a fundamental component for handling big data in internet companies, message queue systems such as Kafka and RabbitMQ have become increasingly popular among backend developers. To meet 360’s internal requirements for durability, scalability, and high availability, the team created a heavily customized Kafka‑based queue called QBus.

QBus Overview

Origin :

LinkedIn’s distributed log processing queue Kafka;

Need to process user behavior (login, browse, click, share, like) and system logs (CPU, memory, disk, network, process status) in real time;

Traditional queues were too heavyweight and slow.

Key Features

Messages are persisted on disk;

High throughput;

Clients maintain consumption state;

Fully distributed producers, consumers, and servers;

Supports subscription model;

Push/pull delivery modes.

Typical Use Cases

QBus can be used as a regular messaging system (synchronous or asynchronous), for log collection and real‑time analysis, data synchronization across data centers via subscription, monitoring service access to prevent uncontrolled crawling, and activity‑stream tracking. It also integrates with Storm and Hadoop, where Storm consumes data and stores results back to QBus, while a copy is asynchronously replicated to Hadoop.

Comparison with Other Queues

Message persistence eliminates loss; Distributed architecture allows seamless scaling without business impact; No single point of failure (multi‑data‑center guarantee); Subscription support enables one‑to‑many delivery across data centers; High throughput; Batch processing of messages; Simple to use.

Architecture Overview

Meta information includes:

Cluster broker IP:port list;

Topic storage locations;

Partition consumption offsets;

All consumer details;

Coordinator node;

Agent configuration and health status.

Message Flow

Broker Cluster

All brokers are peers, enabling smooth horizontal expansion.

Broker Storage

Each partition maps to a logical log composed of multiple segments;

New messages are appended to the last segment file;

Consumers can rewind to a specific offset to re‑consume messages.

High‑Performance Techniques

Leverage OS page cache; user space does not cache messages;

Zero‑copy I/O reduces transmission time by 60‑70% compared with read‑send;

End‑to‑end batch compression;

Sequential disk reads/writes for high efficiency;

Stateless handling of message consumption.

Performance

Supports PHP, C/C++, and Java SDKs. The performance trend with varying message sizes (tested on a single 2.40 GHz CPU core, broker in the same data center) is shown below:

Java SDK Example

Producer (asynchronous):

Consumer (callback mechanism):

Producer (synchronous & asynchronous):

Consumer (callback mechanism):

Improvements Over Kafka

Added reliable transmission request type; brokers acknowledge messages;

Introduced agent nodes for asynchronous message and log collection;

Added coordinator node to handle consumer assignment, mitigating the “thundering herd” problem and reducing multi‑language development cost;

Redesigned PHP SDK and rewrote C/C++ and Java SDKs based on QBus architecture;

Provided a self‑service management portal for easy message administration.