Kafka 2.4.0 Release Summary: New Features, Improvements, and Bug Fixes

Below is a summary of the JIRA issues resolved in Apache Kafka version 2.4.0. For the full documentation, getting‑started guide, and more information about the project, refer to the official Kafka documentation or Chinese Kafka tutorials.

Upgrade notes: before upgrading your cluster, carefully read the upgrade documentation for this version, which covers incompatibilities, breaking changes, performance impacts, and any other changes that may affect Kafka production.

Kafka 2.4.0 includes many important new features. Here are some key highlights:

Allow consumers to fetch from the latest replica.

Support progressive cooperative rebalancing for consumer rebalance protocol.

MirrorMaker 2.0 (MM2), a new multi‑cluster, cross‑data‑center replication engine.

New Java authentication interface.

Support non‑key joins in KTable.

Administrative API for replica reassignment.

New Features (JIRA entries)

[KAFKA-3333] - Support "Always Round‑Robin" partitioning fallback algorithm
[KAFKA-3705] - Support non‑key joins in KTable
[KAFKA-7471] - Multi‑consumer group management (describe, reset, delete)
[KAFKA-7500] - MirrorMaker 2.0 (KIP‑382)
[KAFKA-7800] - Extend Admin API to support dynamic log levels
[KAFKA-8286] - KIP‑460 Admin Leader election RPC
[KAFKA-8447] - New metric for measuring number of tasks on connectors
[KAFKA-8874] - Add optional tag fields to Kafka protocol
[KAFKA-8907] - Return topic configuration in CreateTopics response
[KAFKA-8952] - Fix Jackson‑databind‑2.9.9 vulnerability
[KAFKA-5609] - Connect log4j should log to file by default
[KAFKA-6263] - Expose public metric for group metadata load duration
[KAFKA-6883] - KafkaShortnamer should allow Kerberos principal names to be upper‑cased
[KAFKA-6958] - Allow custom processor names in KStreams DSL
[KAFKA-7018] - Continue using memberId to restart consumer
[KAFKA-7149] - Reduce allocation size to improve KStream scalability
[KAFKA-7190] - WARN about UNKNOWN_PRODUCER_ID when clearing partitions under low traffic
[KAFKA-7197] - Support Scala 2.13.0
[KAFKA-7245] - Deprecate WindowStore #put(key, value)
[KAFKA-7273] - Converters should have access to headers.
[KAFKA-7335] - Store clusterId locally to ensure broker joins correct cluster
[KAFKA-7548] - Paused partition KafkaConsumer should not discard fetched data.
[KAFKA-7772] - Dynamically adjust log level in Connect worker
[KAFKA-7981] - Add Replica Fetcher and Log Cleaner count metrics
[KAFKA-8233] - KIP‑470: Improve TopologyTestDriver test input/output availability
[KAFKA-8305] - AdminClient should support creating topics with default partitions and replication
[KAFKA-8333] - Load high‑watermark checkpoint only once when handling LeaderAndIsr request
[KAFKA-8400] - Do not update follower replica state if log read fails
[KAFKA-8452] - Possibly disable buffer optimization: delete duplicate prior values
[KAFKA-8488] - FetchSessionHandler logs 73 MB allocation in TLAB, may not be operational
[KAFKA-8501] - Remove key and value from exception messages
[KAFKA-8513] - Add kafka‑streams‑application‑reset.bat for Windows
[KAFKA-8544] - Delete old kafka.admin.AdminClient
[KAFKA-8545] - Delete old ZkUtils
[KAFKA-8558] - KIP‑479 – Add StreamJoined overload to KStream #Join
[KAFKA-8559] - PartitionStates.partitionStates causes array growth allocation.
[KAFKA-8560] - Kafka protocol generator should support generic structures
[KAFKA-8563] - Minor: remove networkSend method call, rely on Java vargs boxing/autoboxing
[KAFKA-8594] - Add Kafka Streams compatibility tests for Kafka 2.3.0
[KAFKA-8595] - Support decimal SerDe without hex encoding in JSON
[KAFKA-8601] - Producer improvement: Sticky Partitioner
[KAFKA-8634] - Upgrade ZooKeeper to 3.5.5
[KAFKA-8644] - Kafka protocol generator should allow default values for byte and array fields to be null
[KAFKA-8669] - Add Java security provider in Kafka Security configuration
[KAFKA-8676] - Avoid stopping unnecessary connectors and tasks
[KAFKA-8696] - Clean up Sum/Count/Total metrics
[KAFKA-8717] - Use cached hw/lso offset metadata when reading logs
[KAFKA-8725] - Improve LogCleaner error handling when failing to find the dirtiest log
[KAFKA-8730] - Add API to delete consumer offset (KIP‑496)
[KAFKA-8745] - DumpLogSegments should not hide key when message is null
[KAFKA-8753] - Add JMX metric for number of topics marked for deletion
[KAFKA-8792] - Default ZK config disables AdminServer
[KAFKA-8839] - Improve Kafka logging in debug task lifecycle
[KAFKA-8853] - Create continuous connection test for Trogdor
[KAFKA-8859] - Refactor Streams index cache level
[KAFKA-8880] - Enhance Consumer.committed(partition) to allow multiple partitions
[KAFKA-8884] - Improve class‑cast exception handling in ProcessorsNodes stream errors
[KAFKA-8927] - Delete partition.grouper config and PartitionGrouper interface
[KAFKA-8934] - Introduce instance‑level metrics
[KAFKA-8954] - Incorrect topic existence check in DeleteOffset API
[KAFKA-8956] - Refactor DelayedCreatePartitions#update wait to avoid modifying collection inside foreach
[KAFKA-8984] - Complete label field documentation
[KAFKA-8985] - Use flexibleVersions with LeaderAndIsr, UMR, etc., and increase RequestResponseTest coverage
[KAFKA-9020] - Streams sub‑topology should be sorted by receiver→source relationship
[KAFKA-9033] - Change default client ID in consumer/producer to something more meaningful
[KAFKA-9038] - Allow creating partitions during reassignment
[KAFKA-9058] - Foreign key joins should not require queryable storage

Bug Fixes

[KAFKA-1714] - Better bootstrap for gradle‑wrapper.jar
[KAFKA-5566] - QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied
[KAFKA-5998] - /.checkpoint.tmp not found exception
[KAFKA-6290] - Kafka Connect type conversion should support logical types
[KAFKA-6605] - Flatten SMT does not handle empty fields correctly
[KAFKA-7157] - Connect TimestampConverter SMT does not handle null values
[KAFKA-7263] - Container exception IllegalStateException: selected coordinator has invalid allocation protocol: null
[KAFKA-7849] - Warning when adding GlobalKTable
[KAFKA-7921] - Unstable KafkaStreamsTest
[KAFKA-7937] - ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[KAFKA-7941] - Connect KafkaBasedLog worker thread terminates when offset fetch fails due to broker unavailability
[KAFKA-7990] - KafkaStreamsTest#shouldCleanupOldStateDirs
[KAFKA-7994] - Improve rebalancing and restart stream timing
[KAFKA-8024] - UtilsTest.testFormatBytes fails in German locale
[KAFKA-8041] - LogDirFailureTest#testIOExceptionDuringLogRoll
[KAFKA-8053] - kafka‑topics.sh gives confusing error when topic does not exist
[KAFKA-8078] - TableTableJoinIntegrationTest#testInnerInner
[KAFKA-8086] - GroupAuthorizerIntegrationTest#testPatternSubscriptionWithTopicAndGroupRead
[KAFKA-8104] - Consumer cannot rejoin group after rebalance
[KAFKA-8106] - logValidator reduces ByteBuffer allocation and copying during validation
[KAFKA-8122] - EosIntegrationTest#shouldNotViolateEosIfOneTaskFailsWithState
[KAFKA-8198] - KStreams documentation uses non‑existent method “pipe”
[KAFKA-8262] - MetricsIntegrationTest#testStreamMetric
[KAFKA-8263] - MetricsIntegrationTest#testStreamMetricOfWindowStore
[KAFKA-8297] - Kafka Streams ConsumerRecordFactory generates generic‑related compile errors
[KAFKA-8319] - KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory
[KAFKA-8340] - ServiceLoader fails when using independent plugin path directory
[KAFKA-8392] - Broker leaks metrics when partition leader moves to another node
[KAFKA-8412] - NullPointerException when flushing before closing producer
[KAFKA-8427] - Error cleaning EmbeddedKafkaCluster on Windows
[KAFKA-8442] - Inconsistent ISR output in topic command when using --bootstrap-server
[KAFKA-8448] - Excessive kafka.log.Log logging (memory leak)
[KAFKA-8454] - Add Java AdminClient interface
[KAFKA-8456] - StoreUpgradeIntegrationTest#shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi
[KAFKA-8461] - UncleanLeaderElectionTest#testUncleanLeaderElectionDisabledByTopicOverride
[KAFKA-8487] - Consumer should not reset resetGeneration during REBALANCE_IN_PROGRESS in commit response handling
[KAFKA-8514] - Kafka client should not include Scala Java 8 compatibility library
[KAFKA-8523] - InsertField conversion fails on logical delete events
[KAFKA-8526] - Broker may fail to select new replica directory even when other active directories exist
[KAFKA-8530] - Consumer should handle authorization errors in OffsetFetch
[KAFKA-8550] - Connector validation fails due to alias transformer
[KAFKA-8586] - Source task producer silently fails to send messages
[KAFKA-8602] - StreamThread dies because restored consumer is not subscribed to any topic
[KAFKA-8615] - Change to track partition interval TimestampExtractor
[KAFKA-8620] - Race condition in StreamThread state changes
[KAFKA-8635] - Unnecessary wait when looking up coordinator before transaction request
[KAFKA-8637] - WriteBatch object leaks off‑heap memory
[KAFKA-8649] - Rolling upgrade from Kafka Streams 2.0.0 to 2.1.0 fails
[KAFKA-8662] - Producer fails when previously producing to unauthenticated topic
[KAFKA-8670] - kafka‑topics.sh throws IllegalArgumentException when describing all topics on an empty cluster
[KAFKA-8671] - NullPointerException when topic associated with GlobalKTable changes
[KAFKA-8677] - GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[KAFKA-8688] - Upgrade system test fails due to data loss from old message format
[KAFKA-8698] - ListOffsets response protocol documentation
[KAFKA-8731] - InMemorySessionStore throws NullPointerException on startup
[KAFKA-8736] - Performance: ThreadCache validates empty cache using size()
[KAFKA-8742] - RepartitionWithMergeOptimizingIntegrationTest#shouldSendCorrectRecords_NO_OPTIMIZATION
[KAFKA-8743] - RepartitionWithMergeOptimizingIntegrationTest#shouldSendCorrectRecords_OPTIMIZED
[KAFKA-8748] - DescribeLogDirsRequestTest#testDescribeLogDirsRequest
[KAFKA-8755] - Optimized source table standby task does not write to its state store
[KAFKA-8774] - Connect REST API exposes plain‑text secrets in task endpoint when config value contains special characters
[KAFKA-8782] - ReplicationQuotaManagerTest and ClientQuotaManagerTest should close Metrics objects
[KAFKA-8788] - Optimize client metadata handling for large partitions
[KAFKA-8791] - RocksDB timestamp store should open new store in regular mode
[KAFKA-8801] - Redundant electLeaderForPartitions for some successful elect partitions
[KAFKA-8802] - ConcurrentSkipListMap shows performance degradation in cache and memory
[KAFKA-8804] - Internal Connect REST insecure
[KAFKA-8813] - Race condition when creating topic and changing its config
[KAFKA-8816] - StreamTask indirectly updates RecordCollector offset
[KAFKA-8817] - KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest
[KAFKA-8819] - Converter plugin path does not work as expected
[KAFKA-8824] - InMemoryTimeOrderedKeyValueBuffer propagates null values after config suppression
[KAFKA-8837] - Several integration tests incorrectly use verifyNonDaemonThreadsStatus
[KAFKA-8857] - Config description should not return isReadOnly=false based on synonym
[KAFKA-8860] - SslPrincipalMapper should handle spaces in proprietary names
[KAFKA-8861] - Fix RegexSourceIntegrationTest.testMultipleConsumersCanReadFromPartitionedTopic
[KAFKA-8875] - CreateTopic API should check existence before replication
[KAFKA-8878] - AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS
[KAFKA-8889] - FetchSessionHandler.handleError loses root cause
[KAFKA-8891] - Invalid allocation protocol IllegalStateException: selected coordinator has invalid allocation protocol: null
[KAFKA-8892] - Sorted config display in Kafka Configs help command.
[KAFKA-8911] - Implicit TimeWindowedSerde creates Serde with empty internal serializer
[KAFKA-8943] - Move SecurityProviderCreator to common package
[KAFKA-8945] - ConnectorHealth and AbstractState constructors have incorrect null checks
[KAFKA-8947] - Connect framework incorrectly instantiates REST extension TaskStates
[KAFKA-8950] - KafkaConsumer stops fetching
[KAFKA-8962] - kafka‑admin.AdminClient#describeTopics always goes through controller
[KAFKA-8972] - KafkaConsumer.unsubscribe may cause inconsistent user rebalance callback state
[KAFKA-8974] - Sink connector cannot handle topic list with spaces
[KAFKA-8983] - AdminClient deleteRecords should not unnecessarily invalidate all partitions
[KAFKA-8992] - Do not expose error in RemoveMemberFromGroupResult
[KAFKA-9003] - RepartitionOptimizingIntegrationTest#shouldSendCorrectRecords_OPTIMIZED
[KAFKA-9014] - SourceTask.poll returns empty list causing AssertionError in SourceRecordWriteCounter
[KAFKA-9029] - CooperativeStickyAssignorTest.testReassignmentWithRandomSubscriptionsAndChanges
[KAFKA-9030] - Document instance‑level metrics
[KAFKA-9032] - Use null call when foreignValue is null
[KAFKA-9046] - Connect worker config needs undocumented “admin”. Prefix for DLQ in connector config
[KAFKA-9053] - AssignmentInfo#encode hard‑codes LATEST_SUPPORTED_VERSION
[KAFKA-9069] - AdminClientIntegrationTest.testCreatePartitions
[KAFKA-9073] - StreamThread gets stuck in rebalance state after IllegalStateException
[KAFKA-9078] - System test failure: ConnectRestApiTest
[KAFKA-9079] - System test failure: TransactionsTest
[KAFKA-9080] - System test failure: MessageFormatChangeTest.testCompatibilty
[KAFKA-9084] - AlterPartitionReassignments should not delete all replicas
[KAFKA-9089] - Reassignment should handle unexpected errors
[KAFKA-9133] - LogCleaner thread dies: currentLog cannot be null on unexpected exception
[KAFKA-9140] - Consumer stuck in endless rejoin loop
[KAFKA-9150] - DescribeGroup uses member assignment as metadata
[KAFKA-9156] - LazyTimeIndex and LazyOffsetIndex may cause nio buffer overflow under concurrency
[KAFKA-9171] - DelayedFetch completion may throw exception causing successful produce messages to fail
[KAFKA-9175] - MirrorMaker 2 emits invalid topic‑partition metrics
[KAFKA-9178] - restorePartitions not cleared until last restore task finishes
[KAFKA-9184] - Zombie work re‑joins consumer group, creates redundant tasks and rebalances periodically
[KAFKA-9190] - Server retains connection and opens expired authentication session
[KAFKA-9196] - Segments roll over expose messages before high‑watermark advances
[KAFKA-9198] - StopReplica handler should complete waiting for purgatory operations
[KAFKA-9200] - ListOffsetRequest missing error response for v5
[KAFKA-9203] - kafka‑client 2.3.1 cannot use lz4 compression for topics
[KAFKA-9212] - Continue receiving FENCED_LEADER_EPOCH when sending ListOffsetRequest
[KAFKA-9219] - NullPointerException when polling metrics from Kafka Connect
[KAFKA-9231] - Stream thread may die due to recoverable errors when EOS is enabled
[KAFKA-9244] - Updating old FK reference on RHS should not trigger join result
[KAFKA-9251] - Admin API describing non‑consumer groups hangs forever
[KAFKA-9258] - ConnectorStatusMetricsGroup sometimes NPE
[KAFKA-9261] - NPE when updating client metadata
[KAFKA-9265] - kafka.log.Log instance leaks task when deleting log
[KAFKA-8443] - Allow broker to choose preferred read‑only replica for consumer
[KAFKA-8765] - Delete “unstable” annotation
[KAFKA-9072] - Add Streams Developer Guide for Topology Naming (KIP‑307)

Tests

[KAFKA-9000] - Integration test KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFromRightThenDeleteRightEntity
[KAFKA-9123] - Add system test with large number of partitions

Sub‑tasks

[KAFKA-4222] - Sub‑test QueryableStateIntegrationTest.queryOnRebalance
[KAFKA-4600] - Consumer continues when ConsumerRebalanceListener fails
[KAFKA-8221] - Extend LeaveGroupRequest for batch operations
[KAFKA-8331] - System test for enabling static members on KStream
[KAFKA-8355] - Add static membership to range assignor
[KAFKA-8390] - Replace CreateDelegationToken request/response with auto protocol
[KAFKA-8471] - Replace control request/response with auto protocol
[KAFKA-8492] - Incremental protocol changes to ConsumerCoordinator algorithm (part 2)
[KAFKA-8493] - Add PartitionsLost API to RebalanceListener (part 3)
[KAFKA-8496] - Add compatibility and upgrade path for system tests (part 6)
[KAFKA-8510] - Update StreamsPartitionAssignor to use built‑in sticky partitioning (part 7)
[KAFKA-8539] - Add `group.instance.id` to Subscription class
[KAFKA-8578] - Add functionality to expose RocksDB metrics
[KAFKA-8579] - Expose RocksDB metrics to JMX
[KAFKA-8580] - Compute RocksDB metrics
[KAFKA-8598] - Replace RenewDelegationToken request/response with auto protocol
[KAFKA-8599] - Replace ExpireDelegationToken request/response with auto protocol
[KAFKA-8600] - Replace DescribeDelegationToken request/response with auto protocol
[KAFKA-8603] - File upgrade path
[KAFKA-8609] - Add consumer metrics for rebalancing (part 9)
[KAFKA-8704] - Add PartitionAssignor adapter for backward compatibility
[KAFKA-8760] - KIP‑504: Add new Java Authorizer API
[KAFKA-8848] - Update system tests to use new authentication
[KAFKA-8866] - Make Authorizer create/delete exceptions optional
[KAFKA-8886] - Make Authorizer create/delete asynchronous
[KAFKA-8887] - Use purgatory for async CreateAcls and DeleteAcls
[KAFKA-8902] - Baseline cooperation vs eager rebalancing
[KAFKA-8932] - Merge KIP‑482: add label to TopicConfigErrorCode in CreateTopics response
[KAFKA-8942] - Log RocksDB metrics

Related Reading:

Open‑source Kubernetes‑native CI/CD framework Tekton exploration and usage

Kubernetes Deployment ultimate guide

PPT download – Data challenges in the cloud‑native era, and how MongoDB responds

Kubernetes Job deep dive