Mastering SRE: How to Define SLIs, SLOs, SLAs and Build Reliable Systems

Defining SLIs, SLOs, and SLAs

SRE should involve stakeholders from the start of the software development lifecycle to define SLIs, then provide scientifically‑rigorous SLOs and SLAs once the software is in production, while promoting SLIs awareness among developers.

Key Concepts

SLI : Service Level Indicator – metrics that describe the stability of a service.

SLO : Service Level Objective – the target range for an SLI.

SLA : Service Level Agreement – a contract with users that defines the reliability commitments and remediation actions.

Relationships

The three concepts are interrelated, as illustrated below:

Reliability, Performance, Resilience, Saturation, and Observability

Reliability is measured by availability percentages and downtime calculations (e.g., 99.9% uptime equals 8.76 hours per year). Performance and resilience are trade‑offs; higher performance may reduce resilience and vice‑versa. Saturation refers to the capacity limits of a service, often expressed as CPU usage or request load. Observability relies on signals such as metrics, traces, logs, profiles, and crash data.

Metrics Types

Counter : monotonically increasing values (e.g., request count).

Gauge : values that can increase or decrease (e.g., memory usage).

Histogram : samples observations into configurable buckets (e.g., request latency).

Summary : like a histogram but also provides quantiles.

Tracing

Distributed tracing is essential for modern micro‑service architectures. The preferred protocol is OpenTelemetry.

Causal relationships between Spans in a single Trace
        [Span A]  ←←←(the root span)
            |
        +------+------+ 
        |          |
[Span B]      [Span C] ←←←(Span C is a `child` of Span A)
        |          |
[Span D]      +---+-------+
                |           |
                [Span E]    [Span F]

Temporal relationships between Spans in a single Trace
––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–––––––|–> time

[Span A···················································]
  [Span B··········································]
      [Span D······································]
    [Span C····················································]
          [Span E·······]        [Span F··]

Logging, Profiling, and Crash Analysis

Logs are crucial for observability, especially as workloads shift to Kubernetes. Profiling, enabled by BPF technology, helps tune cloud‑native applications. Crash analysis provides on‑site evidence for post‑mortem investigations.

Error Budget

An error budget is calculated as SLO target + error budget = 100% . Any work that impacts the SLO consumes part of the budget. Exceeding the budget signals potential SLA violations and prompts root‑cause analysis.

Transactional Work Budget

SRE should automate wherever possible: tasks that don’t require human intervention should be handled by machines, and human effort should be limited to areas where automation cannot fully replace it.

Risk Identification and Management

Risk management draws from safety engineering principles such as Hein’s Law, which states that severe incidents are preceded by many minor ones. Understanding potential failure points (e.g., capacity overload, power loss, DNS issues) enables proactive mitigation, including chaos engineering.

Incident Management

Incidents are inevitable; thorough post‑mortems capture causes, background, impact radius, timeline, stakeholders, and remediation steps. A centralized knowledge base for incident reports improves future reliability.

Project Development Work for SRE Teams

Architecture Design : Evaluate both product and SRE platform architectures for reliability and observability.

Software Engineering : Participate in development processes, whether waterfall or DevOps.

Project Management : Often a team member doubles as project manager, balancing schedule, risk, and collaboration.

Testing Process : Emphasize unit‑test coverage for core paths and comprehensive integration testing before release.

CI/CD Pipeline : Build robust DevOps pipelines to maximize delivery efficiency for small SRE teams.

Author: 诸天域 Source: https://blog.zhuxingzhao.com/post/sre-de-zhu-yao-gongzuo/