Understanding Transactions in Apache Kafka: Semantics, API, and Practical Considerations

In a previous post we introduced the various messaging semantics of Apache Kafka, including idempotent producers, transactions, and exactly‑once processing. This article continues that discussion by focusing on Kafka transactions, aiming to familiarize readers with the core concepts required to use the Kafka transaction API effectively.

We cover the main use cases for the transaction API, the semantics of Kafka transactions, details of the Java client API, interesting implementation aspects, and important considerations when using the API.

Transactions are designed for read‑process‑write applications that consume and produce data from asynchronous streams such as Kafka topics, commonly called stream‑processing applications. While early stream processors could tolerate occasional duplicate processing, many modern use cases—e.g., financial debit/credit operations—require strict exactly‑once guarantees.

Exactly‑once processing means that a message a is only considered processed if and only if the resulting message B = F(a) is successfully written, and vice‑versa. Using a regular producer/consumer with at‑least‑once delivery can break this guarantee due to internal retries, reprocessing after crashes, or “zombie” instances that process the same input after a failure.

Kafka solves the second and third problems by making the read‑process‑write cycle atomic through transactions, which provide isolation and atomicity across multiple topics and partitions.

The transactional semantics allow atomic multi‑partition writes: either all messages in a transaction are written, or none are. Offsets are committed only when the transaction is committed, ensuring that consumption and production are treated as a single atomic unit.

Kafka uses a unique transaction.id per transactional producer to prevent zombie instances. The transaction coordinator, running on each broker, manages transaction state stored in an internal transaction log topic. This log holds only the latest state, not the actual messages.

In Java, a transactional producer is configured with a transactional ID and initialized via initTransactions() . Consumers read only committed messages by using the read_committed isolation level. The typical read‑process‑write loop involves starting a transaction, processing records, writing output, sending offset commits, and finally committing the transaction.

The transaction workflow involves four data‑flow phases: (A) producer registers with the coordinator, (B) coordinator updates the transaction log, (C) producer writes data to target partitions, and (D) coordinator runs the two‑phase commit to finalize the transaction.

Practical considerations include choosing a stable transaction.id to avoid zombie producers, understanding the performance impact of transaction overhead (additional RPCs, writes to the transaction log, and per‑partition commit markers), and balancing transaction size against latency.

Performance tests show that transaction overhead is modest and largely independent of the number of messages, but longer commit intervals increase end‑to‑end latency because consumers wait for transaction completion.

Consumers in read_committed mode filter out aborted and in‑flight transactional messages, resulting in no throughput loss compared to non‑transactional consumption.

Further reading links to the original Kafka KIP, design documents, and JavaDocs for deeper details. The conclusion reiterates that Kafka’s transaction API provides the building blocks for exactly‑once stream processing, though full exactly‑once guarantees may require additional handling for external side effects.