How SRE Standards Boost System Reliability in China’s Digital Era

In recent years, frequent outages have made system stability a hot industry topic.

In January 2024 a financial terminal experienced a near‑day‑long outage due to ineffective network redundancy, inadequate monitoring, slow fault localization, poor backup plans, and weak operational capabilities. In April 2024 a cloud service suffered a console and API outage affecting servers, storage, logging, and databases. In August 2024 a music server failure trended on social media, lasting about two hours and possibly linked to a data‑center move. In November 2024 a major e‑commerce platform faced payment failures, transaction creation errors, and service exceptions, causing user inconvenience and financial risk concerns.

With rapid digital technology updates, the importance of information systems has risen, presenting new challenges for stability as system numbers and business scales grow alongside advancing operations technology and evolving operational concepts.

Current information system stability faces several new environmental challenges:

Characteristic One: Large‑scale, distributed systems evolving from monolithic to distributed architectures.

Characteristic Two: High‑frequency changes driven by new business launches and online promotional activities.

Characteristic Three: Complex technology stacks with emerging open‑source tools across OS, middleware, and virtualization.

Characteristic Four: Massive traffic and high concurrency due to rapid mobile internet growth.

The State Council’s Regulation No. 745 on the security protection of critical information infrastructure, effective from 1 September 2021, mandates operators to adopt technical and other necessary measures to prevent security incidents, safeguard stability, and protect data integrity, confidentiality, and availability.

To enhance system stability capabilities, the China Academy of Information and Communications Technology (CAICT) launched a stability initiative in 2020, fully upgrading the “Research‑Development Operations System Reliability and Continuity Engineering (SRE)” framework, which now comprises two major parts: reliability assurance in the development process and reliability assurance on the technical operations side.

1. Development Process Reliability Assurance

This part focuses on reliability measures throughout the software lifecycle, emphasizing design and development, quality assurance, and deployment/release.

Design & Development

Stability admission: Evaluate whether a system meets SRE‑defined production‑ready criteria, covering SLA, metrics, capacity planning, performance measurement, and emergency coordination.

Architecture review: Assess high‑availability, disaster‑recovery, elasticity, and chaos engineering aspects to ensure robust design.

Quality Assurance

Test management: Conduct unit, integration, functional, and performance testing continuously during development.

System quality: Perform code quality checks, reviews, and feedback before merging to the main branch.

Deployment & Release

Release strategy: Define detailed plans including requirements, frequency, methods, and processes, such as gray‑release to mitigate upgrade risks.

Deployment process: Use automation tools to ensure consistent, repeatable deployments, improving availability and maintainability.

2. Technical Operations Reliability Assurance

This part addresses the fault lifecycle, covering prevention, observation, handling, and optimization.

Fault Prevention

Change management: Implement structured change processes to improve quality and reduce risk.

Health checks: Regularly inspect operating environments to detect risks early.

Emergency plans: Establish coordinated response procedures for rapid recovery.

Chaos engineering: Inject faults to test system resilience and automatic recovery.

Performance & capacity: Optimize performance, increase concurrency handling, and apply FinOps principles for cost‑effective resource usage.

Fault Observation

Operational data monitoring: Correlate metrics, logs, and traces to detect, locate, and resolve issues.

Alert management: Use intelligent rules or algorithms for alert aggregation and storm control.

Fault Handling

Fault response: Maintain real‑time monitoring and alerting for swift action.

Fault localization: Combine manual, automated, and AI methods to pinpoint root causes, even amid concurrent failures.

Fault mitigation: Apply throttling, traffic shaping, restarts, rollbacks, scaling, or degradation as appropriate.

Optimization & Improvement

Post‑mortem analysis: Review incidents to prevent recurrence.

Continuous operation: Use objective data to measure stability, refine features, and enhance user experience.

SRE Standard Maturity Levels

The SRE standard defines four maturity levels, each with specific requirements such as adhering to change, configuration, and problem management processes; unified logging and metric evaluation; automation adoption; and established emergency response mechanisms.

Key Recommendations

Define and follow governance processes for change, configuration, and problem management.

Standardize log collection and metric evaluation to ensure trustworthy data.

Introduce automation tools for testing, deployment, and other workflows.

Establish clear incident response responsibilities to shorten recovery time.