How to Define SLIs, SLOs, SLAs and Build Reliable, Observable Systems

Defining SLIs, SLOs, and SLAs

A Service Level Indicator (SLI) is a quantitative metric that reflects the stability of a service. An SLO (Service Level Objective) is the target or acceptable range derived from an SLI. An SLA (Service Level Agreement) is a contract with users that specifies the promised reliability and the remediation actions when the promise is broken.

Relationship between SLIs, SLOs, and SLAs

SLIs provide raw data, SLOs set the acceptable thresholds, and SLAs formalize the commitment.

Reliability, Performance, Resilience, Saturation, and Observability

Reliability metrics

MTTR – Mean Time To Recovery, the average time to restore a service after a failure.

MTBF – Mean Time Between Failures, the average interval between inherent, repairable failures.

MTTF – Mean Time To Failure, the expected time to failure for non‑repairable systems.

Performance vs. Resilience

Maximising performance can reduce resilience, while increasing resilience can degrade performance. The right balance depends on request volume (QPS) and user latency tolerance.

Saturation

Saturation is often expressed as an SLO that measures the capacity a service can handle (e.g., CPU usage). The saturation limit is reached when the service becomes unavailable.

Observability signals

Metrics

Tracing

Logs

Profiling

Crash reports

These signals help assess system stability and accelerate fault detection and resolution.

Metric data model (OpenMetrics)

metric_name{<label name>=<label value>, ...} <value> <timestamp>

node_disk_read_bytes_total{device="sr0"} 4.3454464e+07

node_vmstat_pswpout 0

http_request_total{status="404", method="POST", route="/user"} 94334

Metric types

Counter : monotonically increasing value (e.g., request counts).

Gauge : value that can go up or down (e.g., memory usage).

Histogram : samples observations into configurable buckets (e.g., latency).

Summary : like a histogram but also provides quantiles and total sum.

Distributed tracing

OpenTelemetry is the recommended protocol for tracing in micro‑service architectures. Traces consist of spans with causal and temporal relationships.

Causal relationships between spans in a single trace</code>
[Span A] ←←← (root span)
   |
+------+------+
|             |
[Span B]   [Span C] ←←← (child of Span A)
|             |
[Span D]   +---+-------+
            |           |
        [Span E]    [Span F]

<code>Temporal relationships between spans in a single trace</code>
––|–––––––|–––––––|–––––––|–––––––|–––––––|––→ time
[Span A················································]
   [Span B··································]
      [Span D····························]
    [Span C··········································]
         [Span E·······]   [Span F··]

Logs

Log collection becomes challenging when applications migrate from VMs to Kubernetes due to high volume and distributed nature. Scalable storage and efficient retrieval solutions are required.

Profiling

eBPF enables low‑overhead profiling of cloud‑native applications, facilitating performance tuning.

Crash analysis

Integrating crash reports with other observability signals enables automated detection, reporting, and root‑cause analysis.

Error Budget

The simple formula SLO target + error budget = 100% calculates the remaining error budget. Routine work (releases, changes, incident handling) consumes this budget. Exceeding the budget signals SLA risk and triggers deeper investigation.

Transaction Work Budget

Automate wherever possible.

Use machines to assist where human effort is still required.

If manual tasks dominate, it indicates insufficient automation in the release or change process.

Risk Identification and Management

Hein’s Law states that every serious accident is preceded by many minor incidents and near‑misses. Common risk categories include capacity overload, data‑center power loss, and DNS resolution failures. Documenting these risks and using chaos engineering to simulate failures helps proactive mitigation.

Incident Management

Incidents are inevitable; systematic reporting and post‑mortem analysis prevent recurrence. An incident report should capture:

Root cause (e.g., 2‑4 model)

Background

Impact radius

Timeline

Stakeholders

Resolution process

A thorough post‑mortem reviews symptoms, underlying technical/management issues, extracts lessons, and drives improvements in processes, tooling, alerting, and documentation.

Project R&D Work for SRE

Architecture design

SREs evaluate both product and platform architectures to ensure reliability and observability are built‑in.

Software engineering

Understanding development methodologies (waterfall, DevOps, agile) is essential for contributing to pipelines.

Project management

Often an SRE member doubles as project manager, balancing schedule, risk, manpower, technical analysis, and cross‑team collaboration.

Testing process

Maintain unit‑test coverage for core paths and run integration tests before production releases to catch regressions.

CI/CD pipeline

With small SRE teams, a robust DevOps pipeline is critical to deliver features efficiently and avoid bottlenecks.