Boost Service Availability: MTBF, MTTR, and Practical High‑Availability Tactics

Introduction

The author reflects on personal notes about Google SRE concepts of availability and reliability, written in 2015 and updated with recent case studies. The focus is on practical ways to improve service uptime.

Availability Levels

Availability is expressed as a percentage (SLA). The table below summarizes typical tiers:

Level 1: 90% uptime – ~2.4 hours downtime per day

Level 2: 99% uptime – ~14 minutes downtime per day

Level 3: 99.9% uptime – ~86 seconds downtime per day

Level 4: 99.99% uptime – ~8.6 seconds downtime per day

Level 5: 99.999% uptime – ~0.86 seconds downtime per day

Level 6: 99.9999% uptime – ~8.6 microseconds downtime per day

Generally, a service should aim for at least 99.9% (Level 3) to be considered usable.

MTBF and MTTR

Two industrial‑grade metrics drive availability:

MTBF (Mean Time Between Failures) : how often a component fails.

MTTR (Mean Time To Recover) : how quickly it can be restored.

Availability can be expressed as a function ƒ(MTBF, MTTR). Improving availability means either increasing MTBF or decreasing MTTR, while keeping both realistic.

Practical High‑Availability Measures

Key operational tactics include:

Redundancy (N+2) : Deploy at least two extra instances beyond the minimum required, ensuring service continuity even during simultaneous failures or problematic releases.

Instance Independence : All instances must be equal and isolated; no single point of dependency.

Advanced Traffic Control : Route traffic not just by IP but by business dimensions (API, user type, etc.) to support isolation, quarantine, and protection against “query‑of‑death” attacks.

Change Management

Since releases are the biggest threat to MTBF, the article outlines three essential practices:

Offline Testing : Comprehensive pre‑production testing (unit, integration, performance) to catch issues before they reach production.

Canary/Gray‑Scale Release : Gradual rollout (e.g., 1% → 10% → 100%) with representative user groups, avoiding random or overly slow deployments.

Rollback Support : Design services to allow safe rollback, maintain data compatibility, and provide version‑independent refresh mechanisms to handle configuration or cache failures.

These steps together address 100% of rollback‑related compatibility problems.

7‑Level Availability Chart

The article concludes with a seven‑level diagram describing how services behave under failure, ranging from complete outage with data loss (Level 1) to seamless failover invisible to users (Level 7).