Qbus Service Overview: Architecture, Use Cases, and Implementation Details

Qbus is a cloud‑platform queue service that leverages Kafka as its core component and serves multiple business lines, handling daily traffic of over a petabyte and managing around ten thousand topics.

The service includes surrounding components such as log collection , various language SDKs , persistent storage to HDFS, and custom monitoring, all orchestrated to provide a unified queue solution.

Typical usage scenarios span log systems, tracing services, big‑data pipelines, and stream processing, with primary functions including service decoupling, message communication, and traffic shaping.

Business integration offers three client types: (1) log collection, (2) SDK writes, and (3) reads/writes from other components such as Flink or Hadoop. The first two are provisioned automatically via cloud‑platform tickets, while the third requires manual coordination.

The log‑collection component uses a customized logstash service together with a tailored filebeat version. Users request a topic , specify machines and paths, and the system automatically installs logstash on the target machines. Features include unified configuration management, automatic updates, progress reporting, service self‑check, log‑header enrichment with machine names, and bug fixes; all configurations are stored in the cloud configuration center.

The internally developed SDK, named kafka‑bridge , is open‑source and provides enhancements such as using a VIP instead of a broker list, stronger error handling that skips unavailable replicas, and various parameter optimizations.

For persistent storage, Qbus employs a containerized flume consumer that writes data to HDFS. Configuration resides in the cloud config center and supports dynamic changes. A dynamic adjustment tool monitors consumption status in real time and automatically scales replicas, while historical states are saved for auditability.

The monitoring service, built on Prometheus , forms the foundation for the entire platform. It has been made highly available by splitting processes per business level, uses an InfluxDB cluster for unified backend storage, places an Nginx reverse proxy in front of Prometheus, and adds authentication, thereby delivering stable monitoring data to upper‑layer tools.

In summary, Qbus has achieved fully automated ticketing for most operations, but rapid horizontal scaling and frequent anomalies present new challenges. Future work aims to improve scaling speed, business isolation, and client compatibility, with plans to incorporate pulsar in upcoming versions.