What’s New in Apache Kafka 4.0? A Deep Dive into KRaft, Java 17, and Next‑Gen Consumer Rebalance

From Zookeeper to KRaft: Metadata Management Revolution

Kafka 4.0 is the first major release that removes the dependency on Zookeeper. The new KRaft consensus protocol consolidates metadata management inside Kafka, eliminating external coordination, simplifying deployment, and improving scalability.

Java Support Changes

Java 8 support is dropped; Java 11 is deprecated and no longer supported in brokers, Connect, or tools. Kafka 4.0 requires Java 17 as the minimum runtime, with Java 17 recommended for optimal performance.

Clients (including Kafka Streams) require Java 11 or higher.

Brokers, Connect, and tools require at least Java 17.

Next‑Generation Consumer Rebalance Protocol (KIP‑848)

Traditional eager rebalance caused stop‑the‑world pauses and poor scalability. KIP‑848 moves coordination to the broker’s GroupCoordinator, uses incremental partition assignment, and limits rebalances to affected members, reducing pause time to about one‑third of the previous approach.

Coordination logic shift: Rebalance logic runs on the broker, consumers only send heartbeats.

Incremental assignment: Only partitions of changed members are reassigned.

Fault‑tolerance improvement: Local failures trigger only local rebalances.

Kafka Queues (KIP‑932)

Kafka 4.0 introduces a Share Group mechanism that mimics queue semantics, allowing multiple consumers to process the same partition concurrently for point‑to‑point consumption. It leverages existing topics and partitions with a new consumption and acknowledgment model, but remains an early‑access feature.

Other Important Improvements

Message format v0 and v1 removed: Only v2 is supported, simplifying the protocol stack.

Logging system upgrade: Migration from Log4j 1 (Reload4j) to Log4j 2 (KIP‑1016) improves performance and patches security issues.

Jakarta EE support (KIP‑1032): Kafka Connect upgraded to Jakarta EE and Java EE 10 APIs, compatible with Java 17.

Client API cleanup (KIP‑896): Removal of APIs older than 2.1 to simplify the platform.

Default configuration optimization (KIP‑1030): Adjusted defaults for better out‑of‑box experience.

Leader election safety preview (KIP‑966): Introduces Eligible Leader Replicas to prevent data loss during elections.

For full details, see the official Kafka blog, documentation, and release notes.

https://developer.confluent.io/learn/kraft/

https://cwiki.apache.org/confluence/display/KAFKA/KIP-932+Queues+for+Kafka

https://kafka.apache.org/blog

https://kafka.apache.org/documentation.html

https://dlcdn.apache.org/kafka/4.0.0/RELEASE_NOTES.html