Can Kafka Replace Your Database? Exploring ACID Guarantees with Kafka

Introduction

Kafka can be viewed as an event‑log based storage system that exhibits many ACID‑like properties when used with appropriate patterns. The following sections describe how each property can be achieved with Kafka.

Atomicity

In a traditional single‑node database atomicity is provided by an undo log that rolls back partial writes on failure. With Kafka the same guarantee is obtained by treating the log as the source of truth and using a redo approach:

The application writes each business operation as an immutable event to a Kafka topic.

Each downstream component (e.g., relational DB, cache, search index) runs in its own consumer group and consumes the same ordered stream.

If a consumer crashes, it restarts and resumes from the last committed offset, re‑applying all events that were not yet reflected in its state.

This pattern guarantees that either all components have applied the event or none have, because the event is only considered committed after every consumer group has persisted the offset.

Consistency

Traditional databases enforce integrity constraints (primary keys, foreign keys, check constraints) inside the engine. Kafka does not enforce such constraints automatically; the application must encode the rules in the event processing logic. The event log therefore becomes the single source of truth, and consistency is achieved by ensuring that every consumer validates and applies the same business rules before persisting state.

Isolation

Kafka provides isolation through its partitioning model:

Within a single partition, events are strictly ordered and processed by only one consumer thread at a time, giving serializable isolation for all records that share the same partition key.

Different partitions are independent, allowing parallel processing of unrelated keys.

Example – unique‑name registration:

key = "jane"   // partition key
value = {"action":"register","user":"jane"}
producer.send(topic, key, value);

Two producers send the same key; Kafka guarantees a total order for those two events. The first consumer that processes the event inserts the name into the database; the second consumer sees the existing row and rejects the duplicate, preserving uniqueness without explicit locking.

Lower isolation levels (e.g., read‑committed) combined with a unique index can increase throughput, because the database only needs to enforce uniqueness at insert time while Kafka still provides ordered delivery.

Durability

Kafka achieves durability by replicating each partition across multiple brokers using the In‑Sync Replica (ISR) mechanism:

When a producer writes an event, the leader replica writes to its local log and waits for acknowledgment from all ISR members.

Only after the required number of acknowledgments (configurable via acks=all) is the write considered durable.

If a broker fails, another ISR member is elected as the new leader, ensuring no data loss.

This model is analogous to a write‑ahead log (WAL) in a single‑node database, but the replication is handled transparently by the Kafka cluster.

Practical Limits

While Kafka provides strong guarantees for write‑heavy, event‑driven workloads, it is not a drop‑in replacement for relational databases that require low‑latency point‑queries, complex ad‑hoc joins, or transactional updates spanning multiple unrelated keys. For such scenarios a traditional RDBMS (e.g., MySQL) remains the appropriate choice.