11 Hard‑Earned Lessons from Two Decades of Google Site Reliability Engineering

Google’s Site Reliability Engineering (SRE) program started 20 years ago with a few small data centers, a few thousand servers, and simple Python scripts (e.g., Assigner, Autoreplacer, Babysitter) that managed a basic machine database. Over two decades the compute capacity grew >1,000× and network capacity >10,000×, while operational effort per server fell dramatically.

1. Scale mitigation effort with outage severity

Mitigation actions should match the seriousness of an incident. Over‑reactive fixes such as full‑scale load shedding can cause cascading failures, as seen in a YouTube cache‑configuration change that produced a 13‑minute global outage.

2. Test recovery mechanisms before emergencies

Regularly practice and verify recovery procedures (fire‑drill style) to ensure they meet requirements, are well understood, and can be executed under pressure.

3. Canary all changes

Use progressive rollouts (canary releases) to limit the blast radius of a faulty change. The YouTube cache incident would have been caught early with a canary strategy.

4. Provide a "big red button" for emergency stop

Expose a simple, well‑known mechanism that can instantly halt harmful actions for any critical service.

5. Complement unit tests with integration testing

Unit tests verify individual components but cannot emulate production environments. Integration tests validate that components work together and follow real‑world execution paths, preventing failures like those observed in a Calendar outage.

6. Build redundant communication channels

Relying on a single chat platform (e.g., Google Hangouts/Meet) proved fragile during a massive OAuth token outage. Maintain independent, tested backup channels.

7. Intentionally degrade performance modes

Graceful degradation provides a minimal functional baseline when parts of a system fail, preserving user experience instead of a total outage.

8. Test for disaster resilience and recovery

Disaster‑resilience testing checks that services continue operating under failure; recovery testing ensures systems can return to normal after a shutdown. Table‑top exercises and scenario games help surface "what‑if" questions.

9. Automate mitigations

When clear failure signals appear, automated mitigation can reduce mean time to resolution (MTTR). For example, a multi‑day network incident in 2023 was mitigated faster by auto‑triggered actions.

10. Shorten rollout intervals

Frequent, well‑tested releases lower the chance of large‑scale incidents. A 2022 payment‑system outage was caused by a delayed removal of a database field; shorter intervals would have reduced risk.

11. Avoid single‑point hardware versions

Deploying a single hardware model globally creates a single point of failure. A zero‑day vulnerability in one device type caused a regional outage in March 2020; diversity of hardware mitigates such risk.

These eleven lessons encapsulate two decades of Google’s SRE experience and provide actionable guidance for building resilient, scalable services.