Core Reliability Principles in the Google Cloud Architecture Framework

Google Cloud Architecture Framework explains core principles for running reliable services on the cloud platform. These principles help you reach consensus when reading other parts of the framework, which show how various Google Cloud products and features support reliable services.

Key Terms

The reliability category of the architecture framework uses the following terms, providing essential understanding of how to run reliable services.

Service Level Indicator (SLI)

A Service Level Indicator (SLI) is a quantitative measurement that carefully defines a particular aspect of the service level being provided. It is a metric, not a target.

Service Level Objective (SLO)

A Service Level Objective (SLO) specifies the target level of service reliability. An SLO is the desired value for an SLI. When the SLI meets or exceeds this value, the service is considered “sufficiently reliable.” Because SLOs are the data‑driven basis for reliability decisions, they are the focus of Site Reliability Engineering (SRE) practice.

Error Budget

An error budget is calculated as 100 % – SLO over a period of time. It tells you whether your system is more or less reliable than required within a specific time window and how many minutes of downtime are allowed during that period.

For example, if your availability SLO is 99.9 %, the error budget for a 30‑day period is (1 – 0.999) × 30 days × 24 hours × 60 minutes = 43.2 minutes. Each time the system is unavailable, the error budget is consumed. If the system had 10 minutes of downtime in the past 30 days and starts a new 30‑day cycle with the full 43.2 minutes unused, the remaining error budget drops to 33.2 minutes.

We recommend using a rolling 30‑day window when calculating total error budget and error‑budget‑burn rate.

Service Level Agreement (SLA)

A Service Level Agreement (SLA) is an explicit or implicit contract with your users that defines the consequences when you miss the SLO referenced in the contract.

Core Principles

Google’s reliability approach is based on the following core principles.

Reliability Is Your Primary Feature

New product features may be your short‑term priority, but in the long run reliability is the primary product feature because slow or unavailable products cause users to leave, making other features irrelevant.

Reliability Is Defined By Users

For user‑facing workloads, measure user experience. Users must be satisfied with how your service behaves—for example, measuring request success rate rather than server‑side metrics such as CPU usage. For batch and streaming workloads, you may need to measure data‑throughput KPIs (e.g., rows scanned per time window) instead of server metrics, ensuring timely delivery of daily or quarterly reports.

100 % Reliability Is the Wrong Goal

Your system should be reliable enough to satisfy users but not so reliable that investment becomes unreasonable. Define the required reliability threshold with an SLO, then use the error budget to manage an appropriate rate of change.

Apply the design and operational principles in this framework to a product only when the SLO demonstrates a reasonable cost.

Reliability and Fast Innovation Complement Each Other

Use the error budget to balance system stability and developer agility. The following guidelines help you decide when to move quickly or slowly:

When sufficient error budget is available, you can innovate rapidly and add product features.

When the error budget is depleted, slow down and focus on reliability work.

Design and Operational Principles

To maximize system reliability, the following design and operational principles apply. Each principle is discussed in detail in the reliability category of the architecture framework.

Define Your Reliability Goals

Best practices covered in this part include:

Select appropriate SLIs.

Set SLOs based on user experience.

Iteratively improve SLOs.

Use strict internal SLOs.

Manage development velocity with an error budget.

Build Observability Into Your Infrastructure and Applications

Design principle: Detect your code to maximize observability.

Design for Scale and High Availability

Design principles include:

Create redundancy to improve availability.

Replicate data across regions for disaster recovery.

Design multi‑region architectures to handle regional outages.

Eliminate scalability bottlenecks.

Gracefully degrade service levels under overload.

Prevent and mitigate traffic spikes.

Sanitize and validate input.

Implement fault protection that preserves system functionality.

Design API calls and operational commands to be retryable.

Identify and manage system dependencies.

Minimize critical dependencies.

Ensure every change can be rolled back.

Create Reliable Operational Processes and Tools

Choose good names for applications and services.

Implement progressive deployments with canary testing.

Shift traffic for timed promotions and releases.

Automate build, test, and deployment pipelines.

Prevent operator errors.

Test disaster‑recovery procedures.

Conduct disaster‑recovery drills.

Practice chaos engineering.

Establish Effective Alerts

Optimize alert latency.

Alert on symptoms rather than causes.

Alert on outliers rather than averages.

Establish Collaborative Incident Management Processes

Assign clear service ownership.

Shorten time‑to‑detect (TTD) with well‑tuned alerts.

Shorten time‑to‑mitigate (TTM) with incident‑management plans and training.

Design dashboard layouts and content to minimize TTM.

Document diagnostic procedures and mitigation steps for known incidents.

Use blameless post‑mortems to learn from incidents and prevent recurrence.