Addressing SRE Overload: Causes and Mitigation Strategies

Google's SRE team faces high on‑call load because production incidents often exceed the planned two calls per shift, leading to overload and staff turnover.

To resolve this, the article identifies three primary contributors to on‑call overload: the production environment, alerts, and manual processes.

1. Production Environment

Number of existing problems in production

Introduction of new problems into production

Speed of identifying newly introduced issues

Rate of eliminating or patching vulnerabilities

2. Alerts

Thresholds that trigger paging alerts

Introduction of new paging alerts

Alignment of the service's SLO with dependent services' SLOs

3. Manual Processes

Strict error follow‑up and fix requirements

Quality of data collected for each alert

Monitoring paging load trends

Human changes to the production environment

The article then outlines three scenarios and corresponding actions:

Scenario 1: Existing Bugs – Reduce system complexity, keep dependencies up‑to‑date, perform regular destructive or chaos testing, and run load tests in addition to unit and integration tests.

Scenario 2: New Bugs – Improve testing over time, never neglect load testing, run scenario tests in production‑like environments, use canary releases, and maintain low tolerance for new errors.

Scenario 3: Customer‑Induced Errors – Recognize errors that appear only under specific load levels, request mixes, or unexpected user behavior, and expand testing to cover these intermittent cases.

Finally, the article emphasizes a detection → rollback/fix → forward‑roll strategy, advocating predictable, frequent releases that keep rollback costs low.