How to Ensure Data Consistency in Microservices: Patterns and Pitfalls

Traditional Transaction Management

In a monolithic application a single relational database provides local transactions. The application opens a transaction, performs CRUD operations, and then commits or rolls back. All changes are guaranteed to be atomic and consistent within that single database instance.

Distributed Transactions (2PC/3PC)

When a service accesses multiple data sources a local transaction is insufficient. A Transaction Manager (TM) coordinates the resources using the Two‑Phase Commit (2PC) protocol, which consists of a prepare phase and a commit phase. 2PC can block if a participant crashes and does not guarantee progress, which motivated the Three‑Phase Commit (3PC) protocol that adds an extra pre‑commit phase to reduce blocking.

Microservice Transaction Management and BASE Theory

Traditional distributed transactions do not fit microservice architectures for three main reasons:

Microservices communicate via RPC/HTTP APIs, so a single TM cannot directly manage all resource managers.

Services often use heterogeneous stores (SQL, NoSQL, caches); many NoSQL databases lack ACID transaction support.

Coordinating a large cross‑service transaction would hold locks for a much longer time, severely degrading performance and availability.

Consequently microservices adopt the BASE model (Basically Available, Soft state, Eventual consistency) proposed by eBay’s Dan Pritchett. BASE accepts temporary inconsistency and aims for eventual convergence of data across services.

Consistency Patterns for Microservices

Reliable Event Notification – Synchronous

The primary service performs its business logic and, if successful, sends a message to downstream services within the same call. This couples business code with the messaging system.

public void trans() {
    try {
        // 1. Operate the database
        boolean result = dao.update(data); // throws on failure
        // 2. If DB operation succeeds, send a message
        if (result) {
            mq.send(data); // throws on failure
        }
    } catch (Exception e) {
        rollback(); // revert DB changes
    }
}

Two drawbacks:

Network or server failures after the message is sent may leave the primary service unaware of the delivery status, causing inconsistency.

Strong coupling between business logic and the messaging layer reduces resilience and adds load to the database.

Reliable Event Notification – Asynchronous (Local Event Service)

The primary service writes an event record to a local event table and attempts immediate delivery. If delivery fails, a background worker retries until the event is successfully sent, then removes the record. This decouples the retry logic from the main transaction but still adds write load to the primary database.

Reliable Event Notification – External Event Service

An independent event service stores events after the primary transaction commits (or rolls back). The primary service notifies the event service, which later delivers or discards the events. The event service also polls for undelivered events and queries the primary service for their status, eliminating direct dependency between business and messaging code.

Maximum‑Effort Notification

The primary service attempts to send a message a limited number of times (e.g., three retries). If all attempts fail, the message is dropped and downstream services must provide a query API to recover missing notifications. This approach sacrifices reliability for simplicity and is suitable only for low‑latency, non‑critical notifications.

Business Compensation Pattern

Each downstream service implements a compensation interface. If a downstream call fails, upstream services invoke the compensation actions to roll back previously completed steps. Compensation is usually logical (e.g., marking a record as “canceled”) rather than physically deleting data.

Try‑Confirm‑Cancel (TCC) Pattern

TCC refines compensation by splitting each operation into three explicit phases:

Try : Validate the request and reserve required resources without making permanent changes.

Confirm : After all participants report successful Try, permanently apply the changes.

Cancel : If any Try fails, release the reserved resources.

Example – transferring $100 from Bank A to Bank B:

// Service A (source bank)
try:   UPDATE cmb_account SET balance=balance-100, freeze=freeze+100 WHERE acc_id=1 AND balance>100;
confirm: UPDATE cmb_account SET freeze=freeze-100 WHERE acc_id=1;
cancel:  UPDATE cmb_account SET balance=balance+100, freeze=freeze-100 WHERE acc_id=1;

// Service B (destination bank)
try:    UPDATE cgb_account SET freeze=freeze+100 WHERE acc_id=1;
confirm: UPDATE cgb_account SET balance=balance+100, freeze=freeze-100 WHERE acc_id=1;
cancel:  UPDATE cgb_account SET freeze=freeze-100 WHERE acc_id=1;

TCC provides strong consistency but requires every service to implement both Confirm and Cancel interfaces, increasing implementation complexity.

Comparison of Patterns

The table below summarizes the main characteristics of the four major consistency approaches.