Common Kafka Errors and Their Solutions

1. UnknownTopicOrPartitionException

org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition

Cause: Producer sends messages to a non‑existent topic. Check whether the topic exists or enable auto.create.topics.enable.

2. LEADER_NOT_AVAILABLE

WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Cause: The leader for the partition is unavailable, often because the topic is being deleted or a leader election is in progress. Verify the leader with kafka-topics.sh and ensure brokers are alive; restarting may help.

3. NotLeaderForPartitionException

org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition

Cause: A leader change occurred; the broker you contacted is no longer the leader for the partition. Investigate why the leader moved.

4. TimeoutException

org.apache.kafka.common.errors.TimeoutException: Expiring 5 record(s) for test-0: 30040 ms has passed

Cause: Request timed out. Check network connectivity and consider increasing request.timeout.ms.

5. RecordTooLargeException

WARN async.DefaultEventHandler: Produce request with correlation id 92548048 failed due to [TopicName,1]: org.apache.kafka.common.errors.RecordTooLargeException

Cause: Message size exceeds the broker limit. Reduce batch size or increase message.max.bytes and related limits.

6. Closing socket connection

Closing socket connection to/127.0.0.1.(kafka.network.Processor)

Cause: Mismatch between producer and broker versions can trigger repeated socket‑close errors.

7. ConcurrentModificationException

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi‑threaded access

Cause: KafkaConsumer is not thread‑safe; use a single thread per consumer instance.

8. NetWorkException

[kafka-producer-network-thread | producer-1] o.apache.kafka.common.network.Selector : [Producer clientId=producer-1] Connection with / disconnected

Cause: Network interruption; verify broker network reachability.

9. ILLEGAL_GENERATION

ILLEGAL_GENERATION occurred while committing offsets for group

Cause: Consumer lagged during rebalance; reduce max.poll.records or increase max.poll.interval.ms.

10. advertised.listeners configuration error

java.lang.IllegalArgumentException: requirement failed: advertised.listeners cannot use the nonroutable meta-address 0.0.0.0. Use a routable IP address.

Solution: Set advertised.listeners=PLAINTEXT://{ip}:9092 where {ip} is a reachable address (internal, external, or localhost).

11. PrintGCDateStamps error

[0.004s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:/data/service/kafka_2.11-0.11.0.2/bin/../logs/kafkaServer-gc.log instead.
Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.

Solution: Use JDK 1.8 or a Kafka version ≥ 1.0.x.

12. Producer send failure / consumer cannot consume (kafka 1.0.1)

#(java)org.apache.kafka警告
Connection to node 0 could not be established. Broker may not be available.
# (nodejs) kafka-node异常 (执行producer.send后的异常)
{ TimeoutError: Request timed out after 30000ms ... }

Solution: Verify advertised.listeners and network connectivity (e.g., telnet to the broker).

13. Partitions configuration too small

#(java)org.apache.kafka(执行producer.send)
Exception in thread "main" org.apache.kafka.common.KafkaException: Invalid partition given with record: 1 is not in the range [0...1).
# (nodejs) kafka-node异常 ... { BrokerNotAvailableError: Could not find the leader ... }

Solution: Increase num.partitions for new topics or use kafka-topics.sh --alter --partitions to expand existing topics.

14. Kafka‑Topic operations

Creating a topic:

cd /usr/kafka/bin
./kafka-topics.sh --create --zookeeper master:2181,slave1:2181,slave3:2181/kafka --replication-factor 1 --partitions 1 --topic mobilePhone

Describing a topic:

cd /usr/kafka/bin
./kafka-topics.sh --describe --zookeeper master:2181,slave1:2181,slave3:2181/kafka --topic mobilePhone

Altering partitions (increase only):

cd /usr/kafka/bin
./kafka-topics.sh --alter --zookeeper master:2181,slave1:2181,slave3:2181/kafka --partitions 5 --topic mobilePhone

Deleting a topic requires delete.topic.enable=true in server.properties and removal of Zookeeper paths if necessary.

15. Kafka‑Producer operation

Start a console producer:

cd /usr/kafka/config
./kafka-console-producer.sh --broker-list localhost:9092 --topic newPhone

Ensure listeners is configured (default 9092) and the topic exists.

16. Kafka‑Consumer operation

Start a console consumer:

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic newPhone [--from-beginning]

Use --from-beginning to read from the start of the topic.

17. Kafka startup errors

Lock file error – another Kafka process is using the log directory; kill the stray process.

Permission error on index files – adjust file ownership and permissions.

18. Kafka producer construction failure

org.apache.kafka.common.KafkaException: Failed to construct kafka producer

Root cause often lies in mismatched serviceName between JAAS and Kafka configs. Align serviceName (e.g., kafka) and restart.

19. Consumer offset commit failure

Typical message: Auto offset commit failed for group ... retriable exception. Adjust heartbeat and session timeout settings.

20. Topic creation authentication failure

Authentication failure in Zookeeper

Ensure super.users in server.properties matches the principal in the JAAS file.

21. DataCaptain Kafka component error

Not has broker can connection metadataBrokerList

Fix by setting a routable advertised.listeners address.

22. TimeoutException / Authentication failures across languages

Check server configuration, network connectivity, and use the correct SASL mechanism (ONS for Java, PLAIN for all languages).

23. Leader not available / leader in election

Confirm the topic exists and is of type “Kafka Message”.

24. TOPIC_AUTHORIZATION_FAILED

Ensure the AccessKey has permission for the target topic and consumer group; verify the account hierarchy.

25. Java client SSL close message

Often caused by idle connections being cut in VIP networks; enable retries or suppress the log with

log4j.logger.org.apache.kafka.common.network.SslTransportLayer=ERROR

.

26. Spring Cloud Stream arrayindexoutofboundexception

Set headerMode=raw when consuming messages produced by non‑Spring Cloud clients.

27. “No worthy mechs found” (C++ client)

Install the missing cyrus-sasl-plain package (e.g., yum install cyrus-sasl{-plain}).

28. CID, Consumer ID, Consumer Group and Group ID

All refer to the same concept: a Kafka consumer group. Each CID can subscribe to multiple topics, and each topic can be consumed by multiple CIDs.

29. Viewing consumption progress

Use the ONS console to search for a topic or Consumer ID and view the accumulated message count (total messages minus consumed messages).

30. Message backlog

Usually caused by slow consumption or blocked consumer threads; inspect thread stacks (e.g., with jstack).

31. java.lang.OutOfMemoryError: Map failed

Increase num.recovery.threads.per.data.dir and limit Kafka heap to ≤ 6 GB. Adjust vm.max_map_count (e.g., sysctl vm.max_map_count=262144) to accommodate many topics.