RabbitMQ Livelock: Emergency Fixes for Message Re-delivery Loops

What Is Livelock

Livelock in concurrent programming is a relatively easy concept to grasp. Developers with a few years of experience who have used message queue frameworks have likely encountered or handled similar failures.

Real-World RabbitMQ Livelock Incident

In the described incident, the system used RabbitMQ. Producers sent messages, but consumers failed to process them and did not send an explicit acknowledgment (ACK) back to RabbitMQ. This caused the same message to be re-delivered repeatedly in a vicious cycle: send → receive → fail → requeue → send again.

Outwardly the system appeared to be running and doing work, but it was actually wasting resources without making any progress — essentially spinning in place.

No blocking or deadlock, yet resources are continuously consumed.

Repeated duplicate work, but the task never advances.

How to Handle Livelock in Production

When a real livelock occurs in production, the on-call engineer must act immediately to stop the damage. The author shares two mitigation strategies used during the actual incident.

Fix 1: Emergency Try-Catch with ACK, Then Bug Fix

On the first occurrence, the author deployed two rapid releases:

First release: Wrapped the consumer logic in a broad try-catch, logged the error, and sent an ACK to RabbitMQ so the message would not be requeued.

Second release: Fixed the underlying code bug after proper analysis and testing.

Why two releases? The situation was critical: producers were flooding messages, alerts were firing, and dependent system loads were rising. Any further traffic was meaningless; the only option was immediate stop-the-bleeding. The consumer had to ACK every message regardless of error level.

After the first release, the system stabilized within about 10 minutes.

Why not fix the bug directly? The developer who wrote the code could not diagnose and fix it instantly — they needed time to read code, locate the issue, and possibly involve QA. The system could not wait that long.

What about message loss? That depends on prior design. If messages were not persisted before processing, they are lost. In the author's case, the scenario was a data migration, so re-running the migration was acceptable. For non-loss-tolerant scenarios, the recommendation is to persist messages immediately upon receipt, then process business logic. This allows later replay via scheduled jobs or other tools if a bug surfaces.

Fix 2: "Empty Consumption" Switch

After the incident, the author added a feature flag in the configuration center that could be toggled at runtime. When enabled, the consumer simply returns early without processing:

if (flag == true) {
    return;
}

This "empty consumption" approach is applicable when messages can be lost or when an alternative replay mechanism exists. The author used it in production when a livelock on a shared RabbitMQ cluster (used by multiple teams) threatened other services. Because the consumer had this switch, enabling it resolved the cluster-wide impact immediately.

The article concludes by inviting further questions in the comments.