Consistency Challenges and Solutions in Distributed Systems: CAP, BASE, RPC, and Messaging

This article discusses consistency problems in distributed systems and presents practical solutions based on CAP and BASE theories, RPC handling, and messaging mechanisms.

CAP Theory and BASE Theory – The classic CAP theorem states that a distributed system can at most satisfy two of Consistency (C), Availability (A), and Partition tolerance (P). In practice, most systems choose Availability and Partition tolerance (AP) and rely on BASE (Basically Available, Soft state, Eventual consistency) to achieve eventual data consistency when strong consistency is infeasible.

Consistency Failure Scenarios – A simplified warehouse‑stocking workflow illustrates typical consistency issues: an operator uploads a product, the WMS writes to its database, notifies the central inventory system (SCI), and then notifies the trade service that the product can be sold. Any failure before the SCI update can cause data inconsistency.

Write RPC Example

@Transactional
public void upper(upperRequest request) {
    // 1. Write to warehouse DB
    UpperDo upperDo = buildUpperDo(request);
    wmsService.upper(upperDo);

    // 2. Call RPC to add inventory to SCI
    SciAInventoryRequest sciInventoryRequest = buildSciAInventoryRequest(request);
    sciRpcService.addInventory(sciInventoryRequest);

    // 3. Send message that product can be sold
    TradeMessageRequest tradeMessage = buildTradeMessageRequest(request);
    sendMessageToDealings(tradeMessage);

    // 4. Other processing
    recordLog(buildLogRequest(request));
    return;
}

If any step before the SCI inventory addition fails, the transaction rolls back and no other system is affected, eliminating consistency problems. However, failures after this step can still cause inconsistency.

Message Sending – After a successful write RPC, a message is sent to the trade service. Synchronous messages with retry are commonly used; they rely on broker confirmations to ensure delivery. Example code:

DefaultMQProducer producer = new DefaultMQProducer("ProducerGroup");
producer.setRetryTimesWhenSendFailed(3);
producer.start();
Message msg = new Message("TopicTest", "TagA", "Hello RocketMQ".getBytes());
SendResult sendResult = producer.send(msg);
if (sendResult.getSendStatus() == SendStatus.SEND_OK) {
    log.info("Send Success: " + sendResult);
} else {
    log.warn("Send Failed: " + sendResult);
}

Even with retries, a message may be sent successfully while the subsequent transaction rolls back, leaving the downstream message unrecoverable.

RocketMQ Transaction Messages – Transactional messages provide a two‑phase commit that binds the message to the local transaction, guaranteeing atomicity. This approach is reliable but complex and incurs performance overhead.

Transaction Events + Synchronous Message – Spring’s @TransactionalEventListener can publish domain events after the transaction commits. Listeners then perform RPC calls and send messages, ensuring that messages are only sent after a successful commit.

@Service
public class wmsService {
    @Autowired
    private ApplicationEventPublisher eventPublisher;

    @Transactional
    public void upper(upperRequest request) {
        UpperDo upperDo = buildUpperDo(request);
        wmsService.upper(upperDo);
        UpperFinishEvent upperFinishEvent = buildUpperFinishEvent(request);
        eventPublisher.publishEvent(upperFinishEvent);
        return;
    }
}

@Component
public class upperFinishEventListener {
    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    public void handleUpperFinishEvent(UpperFinishEvent event) {
        SciAInventoryRequest sciInventoryRequest = buildSciAInventoryRequest(event);
        sciRpcService.addInventory(sciInventoryRequest);
        TradeMessageRequest tradeMessage = buildTradeMessageRequest(event);
        sendMessageToDealings(tradeMessage);
        recordLog(buildLogRequest(event));
    }
}

Local Message Table – An alternative is to store pending events in a local message log and process them asynchronously. This reduces the need for complex transaction messages but may introduce latency and resource consumption, especially under high concurrency.

Summary – The article compares three approaches for handling producer‑side consistency: transaction events with normal messages & retry (suitable for low‑real‑time consistency), transaction messages (high consistency, low performance), and local message tables (high reliability, potential latency). Choosing the right method depends on consistency requirements, performance constraints, and system load.