How I Resolved a 1‑Million‑Message Kafka Consumer Backlog

During an afternoon incident the monitoring system reported that the consumer lag for a specific Kafka topic jumped from a few hundred messages to over 1,000,000 and kept rising. The producer rate remained normal (2,000 msgs/s) while the consumer rate dropped from 1,800 msgs/s to 50 msgs/s, causing CPU usage to rise from 30% to over 80% and frequent GC pauses.

1. Incident Symptoms

Key metrics:

Consumer lag: normal < 100, abnormal > 1,000,000

Producer rate: 2,000 msgs/s (unchanged)

Consumer rate: 1,800 msgs/s → 50 msgs/s

Consumer CPU: 30% → 80%+

Consumer GC: normal → frequent

2. Investigation Steps

Step 1 – Check Consumer Logs

// Error log
ERROR com.example.ConsumerService - Failed to process message, skipping retry
java.lang.Exception: Business validation failed
    at com.example.ConsumerService.process(ConsumerService.java:45)

The logs showed many processing failures, but the code always ACKed the message, so failed messages were silently dropped and could not be retried.

Step 2 – Review Consumer Configuration

# Problematic config
spring:
  kafka:
    consumer:
      max-poll-records: 500   # pull 500 records each poll
      fetch-max-wait: 5000    # wait up to 5 seconds
      enable-auto-commit: false  # manual commit
    listener:
      ack-mode: MANUAL

Although manual acknowledgment was enabled, the implementation never called ack.acknowledge() on failure.

Step 3 – Identify the Real Issue

// ❌ No ACK on failure, no retry, no logging of failed messages
catch (Exception e) {
    log.error("Processing failed", e);
    // No ack.acknowledge()
    // No record of the failed message
    // Loop: poll → fail → no ACK → retry → fail again
}

This created a vicious cycle: a batch of 500 messages was pulled, one message failed, the whole batch was blocked, the consumer repeatedly retried the same failing message, and the overall consumption rate fell to zero, leading to massive backlog.

3. Solutions

3.1 Emergency Stop‑Bleed

# 1. Temporarily pause producers (if possible)
# 2. Scale consumer instances
kubectl scale deployment consumer --replicas=10
# 3. Skip problematic messages temporarily (manual ACK in code)

3.2 Long‑Term Fixes

Solution 1 – Proper Exception Handling

@KafkaListener(topics = "order-topic")
public void consume(ConsumerRecord<String, String> record, Acknowledgment ack) {
    try {
        process(record);
        ack.acknowledge(); // ✅ ACK only on success
    } catch (BusinessException e) {
        // Business error: log, send to DLQ, ACK to avoid blocking
        log.warn("Business exception, skipping message: {}", record.value(), e);
        deadLetterService.send(record);
        ack.acknowledge();
    } catch (Exception e) {
        // System error: log and let Kafka retry, but limit retries
        log.error("System exception, retrying", e);
        if (retryCount > 3) {
            deadLetterService.send(record);
            ack.acknowledge();
        }
        // No ACK → Kafka will retry
    }
}

Solution 2 – Use a Dead‑Letter Queue

@Component
public class DeadLetterService {
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
    public void send(ConsumerRecord<String, String> record) {
        // Send to dead‑letter topic
        kafkaTemplate.send("order-topic-dlq", record.value());
        log.info("Message sent to DLQ: {}", record.value());
    }
}

@KafkaListener(topics = "order-topic-dlq")
public void consumeDlq(String message) {
    log.warn("Dead‑letter message requires manual handling: {}", message);
    // Trigger alert for human intervention
}

Solution 3 – Increase Consumer Parallelism

spring:
  kafka:
    listener:
      concurrency: 10   # 10 consumer threads

4. Reusable Checklist

Kafka Consumer Backlog Checklist:
1. Are consumers ACKing messages correctly?
2. Are failed messages handled properly?
3. Is there a dead‑letter queue as a safety net?
4. Is the number of consumer instances sufficient?
5. Are there performance bottlenecks in consumer logic?
6. Is consumer Lag being monitored?

5. Final Thoughts

After the incident the team established rules: every failed message must have a clear handling strategy (retry or DLQ), Lag alerts must trigger immediate notifications, and each topic must be configured with a dead‑letter queue.

If you run Kafka in production, verify that your failure‑handling logic follows these guidelines.