KafkaBridge: A Multi‑Language Kafka Client SDK for Simplified Read/Write Operations

KafkaBridge is an open‑source client SDK that wraps librdkafka to provide a simple, language‑agnostic interface for reading and writing messages to Apache Kafka clusters.

Key Features

Supports multiple languages: C++/C, PHP, Python, Go, with a unified API.

Very few interfaces – only initialization, produce, and cleanup for producers; initialization, subscription, start, and stop for consumers.

Configuration can adjust any librdkafka setting via config files.

Reliability improvements for non‑keyed writes: retries until at least one broker acknowledges.

Both synchronous and asynchronous write modes are available.

Consumer can auto‑commit offsets or allow manual offset commits.

PHP‑FPM optimization reuses long‑lived connections to avoid connection overhead.

Compilation

Depends on librdkafka , liblog4cplus , and boost (header‑only).

C++/C builds with CMake.

Python, PHP, and Go bindings are generated with SWIG.

Each language provides an automatic build script.

Usage – Data Writing

Produce messages by calling the produce interface. In asynchronous mode, the return value indicates whether the internal queue is full; the queue size can be tuned via configuration. In synchronous mode, produce returns success/failure of the write, though performance is lower.

Typical producer API (C++):

bool QbusProducer::init(const string& broker_list, const string& log_path, const string& config_path, const string& topic)

bool QbusProducer::produce(const char* data, size_t data_len, const std::string& key)

void QbusProducer::uninit()

Usage – Data Consumption

Consume messages by subscribing to a topic with subscribeOne (multiple topics are also supported) and then calling start . The consumer runs non‑blocking; each message is delivered via a callback. Users may manually commit offsets via the callback.

Typical consumer API (C++):

bool QbusConsumer::init(const string& broker_list, const string& log_path, const string& config_path, QbusConsumerCallback& callback)

bool QbusConsumer::subscribeOne(const string& group, const string& topic)

bool QbusConsumer::start()

void QbusConsumer::stop()

Performance Test

Cluster: 3 brokers, test topic with 3 partitions.

Producer: single instance, single thread.

No replication: 100 byte message, 1 M messages → 1.7 s. 1024 byte message, 1 M messages → 13 s.

Two‑replica topic: 100 byte message, 1 M messages → 1.7 s. 1024 byte message, 1 M messages → 14 s.

Conclusion

KafkaBridge has been widely used within 360 for services such as search, advertising, IoT, video, security, and gaming, handling petabytes of daily traffic. It is now open‑source (https://github.com/Qihoo360/kafkabridge) and welcomes contributions and feedback.