Stability Engineering Explained: From Entropy Theory to Practical SRE

Why Build Stability

In physics, entropy measures disorder. According to the entropy increase law, a closed system naturally evolves from order to disorder unless external forces intervene. The same principle applies to software systems: a newly released system is orderly (high entropy value), but over time it becomes chaotic and fragile, leading to frequent incidents. To counteract entropy, we must apply stability‑enhancing measures that bring the system back to order.

Significance of Stability Construction

Unstable systems cause real monetary losses. Therefore, stability work is not about increasing revenue but about preventing loss.

Stability Measurement Formula

Availability = MTTF / (MTTF + MTTR)

MTTF (Mean Time To Failure) is the average time a system runs without failure. MTTR (Mean Time To Repair) is the average time to restore service after a failure.

Common Misconceptions

Do not assume a distributed environment is inherently stable.

Avoid deterministic thinking; embrace uncertainty.

Do not shift blame; adopt a sense of ownership.

Industry Status

Internet growth has driven architectural evolution: monolith → vertical applications → distributed → SOA → microservices → service mesh. In modern microservice architectures, stability mechanisms exist at both the application and infrastructure layers.

Infrastructure‑level mechanisms include monitoring CPU load, slow‑SQL detection, MQ backlog alerts, dynamic scaling, and machine health checks.

Current Practices in Stability Governance

“Sprint‑style” stability projects : launch a short‑term stability initiative when incidents surge, then abandon it, causing stability to degrade again.

Point‑wise closed‑loop governance : create dedicated tickets for issues such as slow SQL or rate‑limiting, but risk overwhelming developers and reducing effectiveness.

How to Conduct Stability Governance

Divide stability work into three stages: pre‑vention, mitigation, and post‑mortem.

1. Pre‑vention

Apply techniques like timeout handling, rate limiting, degradation, and slow‑SQL detection to anticipate failures and keep the system operating within design goals.

2. Mitigation (During Incident)

According to the availability formula, reducing MTTR (repair time) and MTTR‑related delays improves SLA. Fast detection relies on monitoring and alerting; rapid resolution requires a clear SOP.

3. Post‑mortem

Focus on learning, not blame. Identify direct causes (what happened) and root causes (why it happened) to prevent recurrence.

Stability Governance Framework

Map governance techniques to the three stages and to product lifecycle phases (early, mid, mature). Examples include:

Capacity testing and auto‑scaling (pre‑vention, mature stage).

Timeout and slow‑SQL handling (pre‑vention, mid stage).

Release checklists, gray‑release, and lossless deployment (change control, early to mature).

Disaster recovery measures such as degradation, isolation, fault injection, multi‑datacenter deployment (post‑vention, mature).

Static code analysis, unit testing, automated testing (engineering quality, early to mid).

Security checks like SQL injection, privilege escalation, anti‑scraping (early stage).

Monitoring & alerting, fault localization, SOPs (during incident).

Specific Governance Plans

For each technique, define a concrete, closed‑loop process. Example – Slow‑SQL governance:

Define a threshold for “slow” queries.

Detect slow queries via monitoring.

Issue a work ticket to the responsible developer.

Validate the remediation.

Example – Timeout governance:

Set appropriate timeout values for each API.

Weekly audit of timeout configurations.

Adjust unreasonable timeouts.

Conclusion

Stability governance is a continuous effort that must be embedded in the development lifecycle. Teams should avoid creating hidden pitfalls and should establish clear standards for aspects such as middleware isolation and timeout configuration, ensuring that stability measures are consistently applied and iteratively improved.