Three Critical Guarantees for Message Queues: No Loss, No Duplicates, No Disorder – Deep Dive into Production‑Grade Solutions

Why the three reliability guarantees matter

When teams first integrate a message queue into a core transaction chain they expect benefits like peak‑shaving, decoupling, and higher throughput, yet production incidents often manifest as missing order status, double‑deducted inventory, delayed payment confirmations, or vanished messages after consumer restarts.

The root cause is usually not the MQ itself but the conflation of four stages: send , persist , consume , and business state transition . A robust system must protect three "lifelines":

Never lose a critical event.

Never process the same business intent twice.

Never break the causal order of a single business entity.

1. No loss – the outbox approach

Relying solely on broker settings such as acks=all cannot prevent loss that occurs before the broker receives the message or after the consumer commits its offset. The production‑grade solution is an outbox table that records the event in the same database transaction as the business data.

CREATE TABLE outbox_event (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  aggregate_type VARCHAR(64) NOT NULL,
  aggregate_id VARCHAR(64) NOT NULL,
  event_type VARCHAR(64) NOT NULL,
  event_key VARCHAR(128) NOT NULL,
  payload JSON NOT NULL,
  status VARCHAR(32) NOT NULL,
  retry_count INT NOT NULL DEFAULT 0,
  next_retry_time DATETIME NOT NULL,
  created_at DATETIME NOT NULL,
  updated_at DATETIME NOT NULL,
  UNIQUE KEY uk_event_key (event_key),
  KEY idx_status_retry (status, next_retry_time)
);

The table tracks four states: PENDING: business transaction committed, event not yet sent. SENT: broker has accepted the event. FAILED: retries exhausted, awaiting manual or automated replay. REPLAYING: the event is being replayed.

Business code writes the outbox record inside a @Transactional method, then a separate dispatcher polls pending rows, sends them to the broker, and updates the status. The dispatcher uses .get() to obtain a definitive send result before acknowledging.

@Component
public class OutboxDispatcher {
    @Scheduled(fixedDelay = 1000)
    public void dispatch() {
        List<OutboxEvent> events = outboxRepository.lockBatchForDispatch(200);
        for (OutboxEvent event : events) {
            try {
                kafkaTemplate.send("order-events", event.getAggregateId(), event.toKafkaMessage()).get();
                outboxRepository.markSent(event.getId());
            } catch (Exception ex) {
                outboxRepository.markRetry(event.getId(), nextRetryTime(event));
            }
        }
    }
}

Broker‑side configuration (e.g., default.replication.factor=3, min.insync.replicas=2, unclean.leader.election.enable=false) guarantees that a successful send truly reached a quorum, but it does not guarantee that the consumer processed the event.

2. No duplicates – idempotent keys and DB constraints

Message‑level idempotence (Kafka's enable-idempotence=true) only prevents duplicate sends from a single producer session. Business‑level duplicates arise from retries, consumer rebalances, or replay jobs. The solution is to separate message_id (for tracing) from a stable idempotent_key that encodes the business intent.

String messageId = "8f1b..."; // broker‑assigned ID
String idempotentKey = "pay:order:1024:success"; // business intent

Consumers store the idempotent key in a dedicated table with a unique constraint, ensuring that the same intent cannot be applied twice.

CREATE TABLE consumer_idempotent_record (
  id BIGINT PRIMARY KEY AUTO_INCREMENT,
  idempotent_key VARCHAR(128) NOT NULL,
  consumer_group VARCHAR(64) NOT NULL,
  topic VARCHAR(128) NOT NULL,
  partition_no INT NOT NULL,
  offset_no BIGINT NOT NULL,
  status VARCHAR(32) NOT NULL,
  created_at DATETIME NOT NULL,
  updated_at DATETIME NOT NULL,
  UNIQUE KEY uk_idempotent_group (idempotent_key, consumer_group)
);

When processing a message, the service first tries to insert the idempotent record; a DuplicateKeyException means the event is a harmless duplicate and can be ignored.

@Transactional
public void handlePaymentSucceeded(PaymentSucceededEvent event) {
    String key = "pay:order:" + event.orderId() + ":success";
    try {
        idempotentRecordRepository.insertProcessing(key, "fulfillment-group");
    } catch (DuplicateKeyException ex) {
        return; // already processed
    }
    fulfillmentService.markPaid(event.orderId(), event.paymentNo());
    idempotentRecordRepository.markDone(key, "fulfillment-group");
}

Redis can be used as a fast front‑door deduplication cache, but the final guarantee must still rely on the database unique constraint.

3. No disorder – ordering per business entity

Global ordering is impractical. The correct approach is to enforce ordering only for events that share the same business key (e.g., orderId, accountId, shopId). This requires a stable partition key and a consumer that processes a partition serially.

kafkaTemplate.send("order-events", order.getOrderId().toString(), event);

Two common pitfalls break ordering:

Temporarily adding a random suffix to the key for load‑balancing.

Using a different key for replay or compensation jobs.

Both destroy the guarantee that all events for the same entity follow the same causal sequence.

For strong ordering, many teams add a business sequence number to the payload and store the last applied sequence per entity.

CREATE TABLE order_event_cursor (
  order_id BIGINT PRIMARY KEY,
  last_applied_seq BIGINT NOT NULL,
  updated_at DATETIME NOT NULL
);

@Transactional
public void apply(OrderEventEnvelope event) {
    long expected = cursorRepository.currentSeq(event.orderId()) + 1;
    if (event.sequenceNo() < expected) return; // duplicate
    if (event.sequenceNo() > expected) throw new RetryLaterException("gap detected");
    orderDomainService.apply(event);
    cursorRepository.advance(event.orderId(), event.sequenceNo());
}

This design automatically discards duplicates, detects gaps, and lets the consumer decide whether to retry later or raise an alarm.

4. Governance – monitoring, replay, and dead‑letter handling

Effective operations require metrics beyond consumer lag. Teams should monitor:

Outbox PENDING and FAILED counts and their age.

Send‑failure rate and retry distribution.

Consumer business‑failure rate.

Dead‑letter queue (DLT) size.

Per‑entity state‑transition latency (e.g., time from ORDER_CREATED to PAID).

Replay success rate and duplicate rate after replay.

Alerting on these signals helps catch the hidden failures that a low lag metric would miss.

5. Choosing the right MQ

Different brokers excel at different aspects:

Kafka – high‑throughput event streams, partition‑level ordering, long‑term retention for replay.

RocketMQ – strong transactional message support, built‑in delay queues.

RabbitMQ – flexible routing, low‑threshold entry, suitable for notification‑type workloads.

Selection should be based on whether the primary need is an event log (Kafka) or reliable command delivery (RocketMQ), and on the team’s ability to maintain outbox, replay, and idempotency infrastructure.

6. Operational pitfalls

Graceful shutdown : In Kubernetes, a pod must finish processing in‑flight messages before termination. Configuration example:

server.shutdown.graceful
spring.lifecycle.timeout-per-shutdown-phase=30s
terminationGracePeriodSeconds: 45

Rebalance handling : When a consumer instance stops, its partitions are reassigned, causing duplicate consumption if offsets were committed before business success. The fix is to commit offsets only after the business transaction completes.

7. Checklist before go‑live

No‑loss checks

Business table and outbox share the same DB transaction.

Outbox rows have explicit status and retry policy.

Failed sends are traceable and replayable.

Acknowledgment occurs after business success.

No‑duplicate checks

Stable idempotent key defined (not just broker message ID).

Idempotent record and business update are in the same transaction.

Duplicate events result in idempotent return, not side effects.

Replay jobs reuse the same idempotent logic.

No‑disorder checks

Partition key is derived from the business entity.

Normal, retry, and replay deliveries use the identical key.

Consumer does not offload processing to an async thread pool that breaks serial execution.

State machine rejects illegal out‑of‑order transitions.

Governance checks

Full‑chain traceability by event_key.

DLT monitoring and handling process.

Load‑test scripts covering rebalance, retry, and replay scenarios.

Separate resource quotas for online traffic vs. compensation traffic.

Conclusion

The three guarantees are not merely broker features; they become business‑state guarantees when combined with outbox persistence, idempotent keys, and strict ordering per entity. A mature message‑queue architecture therefore treats "message delivered" as "business state completed" and provides traceability, compensation, and replay as first‑class capabilities.