Evolution of Apache Kafka Versions and Their Key Features

Recently a colleague asked whether Kafka 0.8.2 can only connect via Zookeeper. Although some users still run the 0.8.x series, it is an old version, and understanding the differences between releases is essential for proper technology selection.

1. Kafka Overview Kafka is a mainstream distributed messaging engine and stream‑processing platform, often used as an enterprise message bus or real‑time data pipeline, and sometimes even as a storage system. Early Kafka focused on high‑throughput messaging, but later versions have evolved into a full‑featured stream processing platform.

Kafka follows the producer‑consumer model: producers send messages to a specific topic partition on a broker, and consumers pull data from one or more partitions. It provides four core APIs – Producer, Consumer, Streams, and Connect – whose support varies across versions.

2. Kafka Versioning Rules Before version 1.0.0 Kafka used a four‑digit version number (e.g., 0.8.2.0). Starting with 1.0.0 it adopted the Major.Minor.Patch scheme, where Major indicates potentially incompatible changes, Minor adds new features, and Patch fixes bugs.

3. Kafka Version Evolution

0.7.x – Very early version with only basic queue functionality and no replication; not recommended for use.

0.8.x – Introduced replication (0.8.0) and a new Producer API (0.8.2). The old and new Producer APIs are:

//旧版本Producer
kafka.javaapi.producer.Producer<K,V>

//新版本Producer
org.apache.kafka.clients.producer.KafkaProducer<K,V>

The new API connects directly to brokers instead of Zookeeper and sends messages asynchronously, offering better performance, but it was still considered unstable at the time.

0.9.x – A major release that added security (authentication, authorization, encryption, Kerberos), a new Consumer API (still connecting to brokers but less stable), and the Kafka Connect component for integrating external systems.

0.10.x – Introduced Kafka Streams, turning Kafka into a distributed stream‑processing platform. Two minor releases (0.10.1, 0.10.2) focused on enhancing Streams. From 0.10.2.2 onward, the Consumer API became stable and Producer performance improved.

0.11.x – A milestone release that brought Exactly‑Once semantics, producer idempotence, and transactional support, which are crucial for reliable stream processing. It also restructured the message format, making it incompatible with earlier versions.

1.x – Continued improvements to Kafka Streams and Connect, added disk‑failure‑tolerant replication, and introduced cross‑directory replica movement for better load balancing.

2.x – Further performance enhancements for Streams and Connect, security hardening, and support for ZStandard compression starting with 2.1.0, which reduces disk usage and network I/O.

4. Version Recommendations

Keep the client and broker versions aligned to avoid compatibility issues.

If you need producer idempotence or exactly‑once semantics, choose version 0.11 or newer.

Select a version you are familiar with that is stable and widely adopted; generally the latest stable release is a safe choice.

The article concludes with references to the original sources and hopes that the summary of Kafka’s version evolution will be useful for practitioners.