When Consumption Looks Successful but DB Remains Empty: ACK, Retry, and Idempotency Explained

What ACK Actually Confirms

ACK tells the message broker whether the delivery can be removed; it only confirms the broker's handling result, not that the database transaction has been committed.

public void consume(Message message) {
    Order order = parse(message);
    transactionTemplate.execute(status -> {
        orderRepository.insert(order);
        return null;
    });
    ack(message);
}

If the ACK is sent before the database transaction, a subsequent DB connection failure, transaction rollback, or process crash leaves the broker believing the message is handled, creating a window where the message is lost but the data is not written.

When the ACK is sent after the transaction, a crash before the ACK or an ACK timeout (while the broker actually received it) can cause duplicate delivery; idempotent logic must handle such repeats.

Kafka commit offsets, RabbitMQ manual acks, and RocketMQ return statuses share this boundary: none can form an atomic cross‑system commit with the business database.

Retry and Idempotency Solve Different Problems

Retry addresses the question “Can the operation be attempted again?” It is suitable for transient failures such as temporary DB unavailability, network glitches, or exhausted connection pools. The consumer should throw an exception or return a failure so the message stays unacknowledged and is redelivered according to the retry policy.

However, retry cannot determine how far the previous execution progressed. If inventory deduction succeeded but the order write failed, a second retry may deduct inventory again, leading to double deduction.

Retry merely adds another execution chance; it does not automatically roll back side effects.

Idempotency addresses “If the same business request runs multiple times, will the result remain correct?” A common approach is to use a stable business idempotent key such as orderId, payment transaction number, or a unique event identifier, and enforce a unique constraint:

insert into consume_record(event_id, created_at)
values (?, now());

If the insert succeeds, processing continues; a unique‑key conflict indicates the event has already been handled, allowing the consumer to acknowledge the message safely.

The idempotent record must be written in the same local transaction as the business write; otherwise the idempotent table could succeed while the business table fails, causing missing data on subsequent retries. Idempotency is not a compensation mechanism: if the consumption never commits, the idempotent key does not restore business data.

Three Common Error Combinations

Automatic ACK may occur at method entry, message fetch, or normal return. If an exception happens after the ACK, the retry path is lost. Swallowing an exception and returning success also cuts off retry, because the framework sees no failure.

try {
    saveToDatabase(message);
} catch (Exception e) {
    log.error("consume failed", e);
}

Therefore, exception classification must influence the ACK decision, not just logging.

Choosing the wrong idempotent key defeats its purpose. Generating a random UUID for each delivery creates a new key each time, and using unstable business fields can mistakenly treat distinct events as the same. The key should derive from the business event itself and stay constant across replays, compensations, and cross‑service propagation.

Defining ACK Timing with Database Commit

Best practice: acknowledge only after the local database transaction commits successfully; roll back or explicitly fail to leave the message unacknowledged for retry.

Retry strategies should distinguish failure types: transient faults get exponential back‑off with a max retry count; deterministic failures (parameter errors, illegal state, parsing issues) should bypass retry and move the message to a dead‑letter queue or manual handling queue.

A dead‑letter queue stores the original message, idempotent key, failure reason, and retry count, enabling teams to decide whether to fix data, fix code, or replay the message.

Even with a committed transaction, two outcomes remain:

Database commit succeeds, ACK fails or times out: the message may be redelivered and requires idempotent handling.

Database commit fails, ACK not sent: the message can be retried according to the retry policy.

This is the normal cost of at‑least‑once delivery semantics.

Cross‑System Scenarios: Filling the Missing Link

If after committing to the database you also need to publish an event (e.g., “order created”), invoking the broker inside the same transaction still risks one side succeeding while the other fails.

When troubleshooting “message consumed but not persisted”, verify in order: early ACK, swallowed consumer exception, database transaction commit, retry trigger, and finally whether the idempotent key is stable and written in the same transaction as the business write.

ACK is not a DB receipt, retry does not guarantee success, and idempotency is not a compensation tool. Identify the transaction boundary that defines business success, then let ACK, retry, and idempotency each handle their respective failure paths.