Backlog Monitoring at 10 Million QPS: From Manual Checks to SLO‑Driven Automation

In a system handling ten million queries per second, message backlog often goes unnoticed for hours, turning a short delay into a massive debt of undelivered messages. This article dissects a six‑stage evolution of backlog monitoring, moving from manual inspection to fully automated, SLO‑driven predictive alerts.

1. Ignored “Silent Alerts”

Many teams equate backlog with lag (the offset difference between production and consumption), but at this scale a single lag metric is insufficient.

1.1 Three Essential Dimensions

A complete view of backlog requires three dimensions:

Lag count : number of unconsumed messages.

Lag time : age of the oldest unconsumed message.

Growth rate : whether lag is increasing, stable, or decreasing.

Relying on lag count alone can be misleading—high‑traffic days naturally produce large counts, while lag time captures short spikes that count misses. Only by combining all three can teams judge whether backlog is abnormal.

1.2 Hidden Nature at Ten‑Million QPS

Because the base rate is huge, a lag of ten thousand messages is a tiny fraction of the total flow and appears invisible on charts, creating a “warm‑water‑frog” effect. Moreover, with hundreds to thousands of partitions, an average lag may look normal while a single partition is severely delayed.

1.3 Cost of Silence

If backlog goes unnoticed, messages may expire, disks can be overwhelmed, downstream state diverges, and recovery may take days, making the loss irreversible.

2. Stage One: Manual Inspection

When traffic is low, teams start with manual checks.

2.1 Command‑Line Checks

kafka-consumer-groups.sh --bootstrap-server xxx \
    --describe --group order-consumer

The output shows CURRENT‑OFFSET, LOG‑END‑OFFSET, and LAG, giving an instant view of backlog. Similar tools exist for RocketMQ and Pulsar.

2.2 Excel Inspection Sheets

As volume grows, engineers copy lag numbers into spreadsheets to compare against previous days, forcing regular human review.

2.3 When Manual Stops Scaling

Beyond ~50 topics, 500 partitions, or 100 consumer groups, manual review becomes impractical; even doubling staff only halves response latency, and night‑time coverage is unreliable.

3. Stage Two: Threshold Alerts

Teams add simple static thresholds to alert when lag exceeds a fixed value.

3.1 Basic Rule

When consumer_group_lag > 100000 for 5 minutes, trigger an alert.

In practice, static thresholds either fire too often or miss incidents because traffic fluctuates between peak and off‑peak periods.

3.2 The Threshold Dilemma

High‑traffic periods require higher thresholds, while low‑traffic periods need lower ones; promotional spikes can also break static settings, leading to years of “threshold‑tuning” pain.

3.3 Value of Static Thresholds

Static thresholds still serve two purposes:

Absolute floor : e.g., lag > 10 million or consumption stopped > 10 minutes must always alert.

New service bootstrapping : wide static limits protect until enough historical data exists for dynamic models.

Thus static alerts act as a safety net rather than the sole mechanism.

4. Stage Three: Multi‑Dimensional Monitoring

Adding more dimensions mitigates the shortcomings of single‑metric thresholds.

4.1 Lag Time Over Lag Count

Lag time (age of the oldest message) is volume‑independent and directly reflects user impact; for ten‑million QPS systems, broker‑side “oldest unconsumed message age” is preferred.

4.2 Partition‑Level Visibility

Monitoring each partition uncovers isolated issues that average metrics hide; a single lagging partition can cause severe downstream problems even when total lag looks normal.

4.3 Derived Rate Metrics

Lag growth rate : increase per unit time, indicating a disaster‑prelude.

Catch‑up ETA : estimated time to clear current lag at the present consumption rate.

Example: 1 million lag with a consumption rate of 50 k/s clears in 20 seconds, whereas 300 k lag at 1 k/s needs 5 minutes—crossing the SLO.

4.4 Composite Alert Logic

lag count > 10 000 AND lag time > 30 seconds AND growth rate > 0 for 1 minute → alert.

This three‑dimensional rule filters out transient spikes and forms the “ultimate” static‑threshold evolution before moving to model‑based detection.

5. Stage Four: Anomaly Detection

Static rules cannot adapt to traffic changes; statistical anomaly detection fills the gap.

5.1 Same‑Period and Rolling‑Period Comparison

Compare current lag to the same time yesterday (同比) or the past 30 minutes (环比); alert when deviation exceeds N × standard deviation.

5.2 STL Seasonal Decomposition

Decompose lag into trend, seasonality, and residual components; use residuals for alerts, a technique employed by Twitter and Netflix.

5.3 When to Use Machine Learning

Complex models (LSTM, Transformer) often underperform simple rules; a pragmatic path is to solidify static and multi‑dimensional alerts first, then add STL, and finally reserve ML for the remaining ~5 % of extreme cases.

6. Stage Five: Predictive Monitoring

Even anomaly detection reacts after the fact; predictive monitoring aims to warn before lag reaches danger.

6.1 Linear Extrapolation

If (current lag + growth_rate × 10 minutes) > emergency threshold → early alert.

This works well for linear degradation scenarios such as gradual consumer OOM.

6.2 Capacity Forecasting

Combine current lag with consumption capacity to estimate when lag will be cleared; if ETA exceeds the business SLO, trigger an alert and optionally auto‑scale or throttle.

6.3 Practical Tips

Keep the prediction model simple and explainable (linear extrapolation covers ~90 % of cases).

Label alerts as “predictive” to differentiate from actual threshold breaches.

7. Stage Six: SLO‑Driven Monitoring

Link low‑level metrics to business impact via Service Level Objectives.

7.1 From Lag to Business SLO

Users notice delays: 3 seconds feels OK, 30 seconds triggers complaints, 5 minutes feels like a outage. Monitoring should therefore align alerts with these user‑perceived thresholds.

7.2 Error Budget and Alert Prioritization

P99 end‑to‑end latency < 5 seconds → monthly error budget 0.1 % (≈43 minutes). When 90 % of the budget is consumed, raise alert severity.

7.3 Tiered Response

SLO‑driven alerts move from binary on/off to graded responses, ensuring only critical backlog wakes on‑call engineers while all incidents are recorded for analysis.

8. Evolution Path Summary

The six stages form a clear roadmap that should not be skipped; each stage teaches the team what to observe, what constitutes an anomaly, what defines health, what is “normal” historically, how far ahead to intervene, and finally what truly matters to the business.

Tool stacks evolve accordingly, from simple command‑line utilities and spreadsheets to open‑source monitoring stacks (Prometheus, InfluxDB, etc.) and eventually custom platforms for the highest scale.

Even with full automation, on‑call engineers remain the indispensable “last mile” to interpret alerts and drive remediation.