How Kafka’s OffsetIndex and TimeIndex Optimize Message Retrieval

Kafka Index Files Overview

Kafka stores two types of index files for each log segment: .index (the offset index) and .timeindex (the timestamp index). Each index type maps different key‑value pairs to enable fast location of messages.

1. OffsetIndex – Offset Index

1.1 Definition

The OffsetIndex maps a relative offset (K) to the physical file position (V) of the first byte of a message within a log segment.

Structure

K – the relative offset of the message.

V – the physical file position of the message’s first byte.

Entry Size

The abstract method entrySize defines the byte size of a single K‑V pair. For OffsetIndex this size is 8 bytes: 4‑byte integer for the relative offset and 4‑byte integer for the file position.

Kafka’s full offset is a 64‑bit long (8 bytes), but OffsetIndex stores only the difference from the segment’s base offset, allowing each entry to occupy only 4 bytes for the offset component, saving roughly 4 MiB per 1 000 entries.

Reading an Entry

When a consumer reads from a specific offset, Kafka uses the OffsetIndex to locate the physical file position directly, avoiding costly sequential reads.

The method parseEntry constructs an OffsetPosition object containing the K and V values for the entry.

Appending Entries

To append a new entry, Kafka writes the relative offset and the physical file position into the memory‑mapped file (mmap).

Truncation

The truncateToEntries operation can cut the index file to retain only a prefix of entries (e.g., keep the first 40 of 100 entries).

Lookup

The lookup method returns the greatest offset not exceeding a target offset and its file position, effectively acting as a floor function for offsets.

2. TimeIndex – Timestamp Index

2.1 Definition

The TimeIndex maps a timestamp (stored as a long) to a relative offset (stored as an integer). Each entry therefore occupies 12 bytes: 8 bytes for the timestamp and 4 bytes for the relative offset.

Writing Entries

The maybeAppend method writes the timestamp and relative offset to the mmap, ensuring both values increase monotonically. Non‑monotonic timestamps cause consumer confusion because they may retrieve incorrect offsets.

Monotonicity Requirement

If a timestamp that is older than the previous one is written, consumers that filter by timestamp may read stale or wrong data.

3. Summary and FAQ

OffsetIndex and TimeIndex are used together: first the TimeIndex finds the offset for a given timestamp, then the OffsetIndex locates the physical file position for that offset. Both indexes share the broker configuration log.index.size.max.bytes.

Important caution: Do not rename, delete, or modify index files manually; doing so can cause broker startup failures. Kafka can rebuild indexes, but arbitrary deletions are risky.

During partition initialization, Kafka pre‑allocates about 10 MiB for each index file, which may appear empty.

Consumer offsets are stored per consumer group ( <groupId, topicPartition, offset>). New consumers start based on the auto.offset.reset policy if no prior offset exists.

Kafka does not provide a built‑in delayed‑message feature; it must be implemented by the application.