Ensuring Data Consistency in Microservices: From CAP to Transactional Messaging

Introduction

The classic CAP theorem states that a distributed system can simultaneously guarantee at most two of consistency (C), availability (A), and partition tolerance (P). In monolithic applications strong consistency is achieved via local ACID transactions, but microservice decomposition often breaks atomicity, creating consistency challenges that business logic still requires to solve.

CAP Theory and BASE Theory

CAP forces a trade‑off; most systems choose availability (A) and partition tolerance (P), sacrificing strict consistency. The BASE model (Basically Available, Soft state, Eventual consistency) complements CAP by allowing temporary inconsistencies while ensuring that data eventually converges to a consistent state.

Consistency Failure Scenarios and Solutions

1. Write RPC

When a service invokes a write RPC to update another service (e.g., adding inventory to a central system), failures can leave data inconsistent. Solutions include:

Retry with idempotent operations.

Local message table to persist the intent and retry asynchronously.

Read RPCs do not affect other services, while write RPCs do.

2. Message Sending

After a successful write RPC, a message is sent to downstream services. Synchronous messages with retry provide higher reliability than pure asynchronous messages. RocketMQ offers both synchronous messages and transactional messages that bind a two‑phase commit to the local transaction.

@Transactional
public void upper(upperRequest request) {
    // 1. Write to warehouse DB
    UpperDo upperDo = buildUpperDo(request);
    wmsService.upper(upperDo);
    // 2. Call RPC to add central inventory
    SciAInventoryRequest sciInventoryRequest = buildSciAInventoryRequest(request);
    sciRpcService.addInventory(sciInventoryRequest);
    // 3. Send sell‑able message
    TradeMessageRequest tradeMessage = buildTradeMessageRequest(request);
    sendMessageToDealings(tradeMessage);
    // 4. Other processing
    recordLog(buildLogRequest(request));
    return;
}

RocketMQ synchronous send example:

DefaultMQProducer producer = new DefaultMQProducer("ProducerGroup");
producer.setRetryTimesWhenSendFailed(3);
producer.start();
Message msg = new Message("TopicTest", "TagA", "Hello RocketMQ".getBytes());
SendResult sendResult = producer.send(msg);
if (sendResult.getSendStatus() == SendStatus.SEND_OK) {
    log.info("Send Success: " + sendResult);
} else {
    log.warn("Send Failed: " + sendResult);
}

Transactional messages guarantee atomicity between the message and the local transaction but add complexity and performance overhead.

3. Local Message Table

Persisting pending events in a local message log enables asynchronous retries and decouples the downstream call from the primary transaction. Advantages include high reliability and simplicity; drawbacks are latency, potential data backlog, and increased resource usage in high‑concurrency scenarios.

Not suitable for very high‑throughput workloads due to added delay.

Long‑running transactions can cause large transaction scopes.

Message payload size may become a storage concern.

Conclusion

The article compares three main approaches for handling producer‑side consistency in distributed systems:

Transactional event + normal message & retry : suitable for low‑latency consistency requirements, high concurrency, simple implementation but requires manual idempotency handling.

Transactional messages : best for high‑consistency scenarios such as financial transactions; ensures atomicity but incurs performance penalties and implementation complexity.

Local message table : fits cross‑service transactions and eventual consistency; offers reliability with asynchronous retries but may introduce latency and resource consumption.

In practice, the author’s team prefers the local message table approach for its balance of reliability and simplicity, especially when write‑RPC failures are the primary source of inconsistency.