How Deleting a Kafka Topic Removes Consumer Offsets and Why It Matters

Problem Overview

When a Kafka topic is deleted and later recreated, does a new consumer with the same group ID start from the previously committed offset stored in __consumer_offsets or from the position defined by its auto.offset.reset setting?

Reproduction Steps

Step 1: Create a topic named topic-offset.

Step 2: Produce several messages to the topic and close the producer.

Step 3: Start a consumer in group group.offset, consume the messages, and commit the offset to __consumer_offsets, then close the consumer.

Step 4: Delete the topic-offset topic.

Step 5: Within the offset retention period (default 1440 minutes), recreate topic-offset with the same configuration.

Consumer Offset Storage

In older Kafka versions offsets were stored in Zookeeper, which is unsuitable for high‑write workloads. Modern consumers store offsets in the internal topic __consumer_offsets. When a consumer commits an offset, it sends an OffsetCommitRequest to the GroupCoordinator, which writes the offset to __consumer_offsets and keeps an in‑memory copy. After a broker restart, the coordinator loads all offset records from __consumer_offsets back into memory.

Effect of Topic Deletion

Although offsets appear persistent, deleting a topic also removes any offset records that reference that topic. The deletion workflow performed by the Kafka controller includes:

Removing the Zookeeper nodes /admin/delete_topics/topic-offset, /brokers/topics/topic-offset, and /config/topics/topic-offset.

Marking the log files of the topic on each broker for deletion.

Instructing each broker’s GroupCoordinator to delete offset metadata for the topic. The coordinator scans all consumer groups, deletes partition entries for topic-offset, and writes a tombstone (a record with a null value) to __consumer_offsets. Because the topic’s cleanup policy is compact, the tombstone eventually removes the offset record.

Conclusion

After the original topic-offset is deleted, all associated offset entries are also removed. Consequently, a new consumer created after step 5 will not find any previous offset and will start according to its auto.offset.reset configuration (earliest or latest).