How MQ Message Reordering Triggered a Major Business Outage

Incident Overview

During a night shift the logistics system reported that many orders were marked as paid while the corresponding shipment remained in a "pending payment" state. The on‑call engineer rolled back the recent change and began manual data reconciliation.

Root Cause Analysis

The original workflow publishes an order_created_event when an order is created and an order_paid_event after payment. Both events are consumed by the logistics service, which creates a shipment on the creation event and updates it to "ready for shipment" on the payment event.

A new "gift‑giving" feature rewrote the flow: the application first creates the order, then immediately simulates a successful payment so that a gift can be attached without changing downstream contract messages. The two events are sent to separate RocketMQ topics ( order_created_event and order_paid_event).

Because the topics are independent, RocketMQ does not guarantee delivery order. In the normal purchase flow the five‑second delay between creation and payment naturally preserves order, but the simulated payment sends the two events almost simultaneously. Consequently the consumer may receive the payment event before the creation event, causing an inconsistent state.

Impact of Reordered Messages

If the payment event arrives first, the logistics service cannot find a corresponding shipment record and the update fails. When the creation event arrives later, it creates a shipment in "pending payment" status, but the earlier payment event has already been missed, leaving the shipment stuck.

Mitigation Strategies

Active Delay (Timer‑Based)

The simplest fix is to delay the payment‑success message. A naïve Thread.sleep blocks threads and degrades latency, so a timer‑based approach is preferred. The timer registers a ScheduledExecutorService task that publishes the payment event after a configurable delay (e.g., five seconds):

@TransactionalEventListener
public void handle(OrderPaidEvent event) {
    Runnable task = () -> {
        rocketMQTemplate.convertAndSend("order_paid_event", event);
    };
    executor.schedule(task, 5, TimeUnit.SECONDS);
}

Drawbacks of this in‑memory timer solution:

Tasks exist only in memory; an ungraceful shutdown discards them, causing lost business logic.

Asynchronous execution can give a false sense of completion.

Large task volumes may exhaust memory and trigger OOM.

Ordered Messages (Partitioned Ordering)

RocketMQ offers two ordered‑message modes:

Global ordering : all messages share a single queue, limiting scalability.

Partitioned ordering : messages with the same sharding key are routed to the same queue, preserving order within each partition.

Using partitioned ordering, the order ID is set as the sharding key so that both creation and payment events for the same order are consumed in order:

@TransactionalEventListener
public void handle(OrderCreatedEvent event) {
    Long orderId = event.getOrderId();
    Message<OrderCreatedEvent> message = MessageBuilder.withPayload(event)
        .setHeader(RocketMQHeaders.KEYS, orderId) // Sharding Key
        .setHeader(RocketMQHeaders.TAGS, "OrderCreatedEvent")
        .build();
    rocketMQTemplate.send("order_event_topic", message);
}

@TransactionalEventListener
public void handle(OrderPaidEvent event) {
    Long orderId = event.getOrderId();
    Message<OrderPaidEvent> message = MessageBuilder.withPayload(event)
        .setHeader(RocketMQHeaders.KEYS, orderId) // Sharding Key
        .setHeader(RocketMQHeaders.TAGS, "OrderPaidEvent")
        .build();
    rocketMQTemplate.send("order_event_topic", message);
}

Reference Implementation

Repository: https://gitee.com/litao851025/learnFromBug

Source directory: https://gitee.com/litao851025/learnFromBug/tree/master/src/main/java/com/geekhalo/demo/mq/disorder