Boost Your Kafka Integration with KafkaBridge: Multi-Language SDK Overview

Introduction

KafkaBridge encapsulates read and write operations to a Kafka cluster with a minimal, easy‑to‑use, stable, and reliable interface, supporting C/C++, PHP, Python, and Go, and includes optimizations for long‑connection reuse in php‑fpm scenarios, widely used within 360.

Background

Kafka is a leading distributed streaming platform in the big‑data field, originally developed by LinkedIn and donated to Apache in 2010.

Within 360, Kafka serves search, advertising, IoT, video, security, gaming, and other core services, handling roughly 1.2 PB of writes and 2.4 PB of reads daily.

Although the official Java client SDK exists, 360 needed a multi‑language solution, leading to the development of KafkaBridge.

Features

Supports multiple languages (C/C++, PHP, Python, Go) with a unified interface.

Few interfaces, simple to use.

Advanced users can configure all librdkafka settings via configuration files.

Attempts to write every message successfully even without key‑based partitioning.

Provides both synchronous and asynchronous write modes.

Allows manual offset commits during consumption.

Optimized for php‑fpm by reusing long connections to avoid frequent connect/disconnect overhead.

Compilation

Depends on librdkafka, liblog4cplus, and Boost (only header files needed).

C++/C builds with CMake.

Python, PHP, and Go bindings are generated with SWIG.

Each language includes automatic build scripts for easy compilation.

Data Writing

In non‑keyed writes, the SDK retries until at least one broker acknowledges the message.

Writing requires a single

produce

call; in asynchronous mode, the return value indicates queue status, configurable via a config file.

In synchronous mode,

produce

returns success status but may increase CPU usage; asynchronous mode is recommended for performance.

// Initialization interface
bool QbusProducer::init(const string& broker_list, const string& log_path, const string& config_path, const string& topic);
// Write data interface
bool QbusProducer::produce(const char* data, size_t data_len, const std::string& key);
// Cleanup interface (must be called)
void QbusProducer::uninit();

Data Consumption

Subscribe to a topic (or multiple topics) via

subscribeOne

, then start consumption with

start

; consumption is non‑blocking and delivers messages through a callback.

Supports manual offset commits via the callback.

// Initialization interface
bool QbusConsumer::init(const string& broker_list, const string& log_path, const string& config_path, QbusConsumerCallback& callback);
// Subscribe to a topic
bool QbusConsumer::subscribeOne(const string& group, const string& topic);
// Start consumption
bool QbusConsumer::start();
// Stop consumption
void QbusConsumer::stop();

Performance Test

Test environment: three‑broker Kafka cluster, test topic with three partitions, single‑threaded producer.

Without replication:

With two‑replica configuration:

Conclusion

KafkaBridge is actively used within 360 and has been open‑sourced. Users are encouraged to provide feedback and contribute to its ongoing improvement. The source code is available at https://github.com/Qihoo360/kafkabridge .