KafkaBridge: A Multi‑language Kafka Client SDK for High‑Performance Data Production and Consumption

KafkaBridge encapsulates read and write operations to a Kafka cluster with a very small, easy‑to‑use interface, supporting C/C++, PHP, Python, and Go, and includes special long‑connection reuse optimizations for php‑fpm; it is already widely used inside 360.

Kafka has become the most popular distributed streaming platform in the big‑data field, originally developed by LinkedIn and donated to Apache in 2010. 360 uses Kafka across search, advertising, IoT, video, security, and gaming, handling roughly 1.2 PB of writes and 2.4 PB of reads daily, which motivated the creation of a language‑agnostic client SDK.

Built on top of librdkafka , KafkaBridge abstracts many low‑level details, exposing a unified API across languages, requiring only a few calls to produce and consume messages, while improving reliability and providing configuration options for advanced users.

Compilation depends on librdkafka , liblog4cplus , and boost . C++/C builds use CMake, while Python, PHP, and Go bindings are generated with SWIG; each language ships with automatic build scripts.

Data Writing – Production is performed via a single produce call. In asynchronous mode the return value indicates queue saturation, and the queue size can be tuned via configuration. In synchronous mode the call returns success status but incurs higher CPU usage; asynchronous mode is generally recommended. The core producer API is:

bool QbusProducer::init(const string& broker_list, const string& log_path, const string& config_path, const string& topic);
bool QbusProducer::produce(const char* data, size_t data_len, const std::string& key);
void QbusProducer::uninit();

Data Consumption – Consumption starts by subscribing to a topic with subscribeOne , then calling start . The SDK runs non‑blocking and delivers each message via a callback; users may manually commit offsets if desired. The consumer API (C++ example) is:

bool QbusConsumer::init(const string& broker_list, const string& log_path, const string& config_path, QbusConsumerCallback& callback);
bool QbusConsumer::subscribeOne(const string& group, const string& topic);
bool QbusConsumer::start();
void QbusConsumer::stop();

Performance Tests – Tests were run on a three‑broker Kafka cluster with a single‑threaded producer. Without replication, 1 million 100‑byte messages were sent in 1.7 seconds and 1 million 1024‑byte messages in 13 seconds. With a replication factor of 2, the same workloads took 1.7 seconds and 14 seconds respectively, demonstrating low overhead.

KafkaBridge is now open‑source (https://github.com/Qihoo360/kafkabridge). The project welcomes feedback and contributions from the community.