The article explains why microservice governance—through measurement, targeted remediation, and verification—is crucial for maintaining system stability, reducing complexity, and improving availability in large‑scale, rapidly evolving architectures.
This article provides a comprehensive SRE guide covering senior role definitions, monitoring objectives and implementation, core metric selection, link and event monitoring, capacity planning and mitigation strategies, a real‑world health‑code outage case, and change‑management best practices to improve reliability and efficiency.
This comprehensive guide explains the origins, methodologies, and practical steps of fault postmortems—including PDCA, GRIA, aviation safety lessons, industrial accident theory, and software reliability metrics—to help teams systematically investigate incidents, derive actionable improvements, and continuously enhance system availability.
This article explains the essence, purpose, and step‑by‑step process of fault postmortems—including preparation, root‑cause analysis, improvement actions, and decision making—while covering PDCA and GRIA methodologies, industry examples, MTTR/MTBF metrics, and practical templates for lasting reliability.
This article summarizes a technical talk on the challenges of cloud‑native transformation, the design of an application‑centric observability platform using CMDB, Prometheus, Thanos and VictoriaMetrics, practical solutions for high‑cardinality metrics and alerting, and future directions such as eBPF and AI‑driven fault detection.
The article introduces the Google SRE Workbook as a practical supplement to the original SRE book, explains the five core SRE foundations—including SLO, SLI, SLA, monitoring, and real‑world case studies from Google and Kingsoft Office—while also promoting an upcoming SRE‑DevOps live session.
This article examines the challenges of traditional on‑call duty systems for operations teams, proposes an SRE‑inspired rotation model that involves developers, defines concrete KPI targets, and describes how automation and chat‑bot tools can streamline incident response and reduce internal friction.
The article recommends two essential Google SRE books for backend developers, explains what SRE is, how it differs from traditional operations, and shows how the concepts like SLI/SLO, incident postmortems, and reliability engineering can be applied to improve system availability and stability.
AlterShield is an open‑source, end‑to‑end change‑control platform that systematizes change perception, risk analysis, and defense across distributed cloud‑native environments, enabling SRE teams to mitigate stability risks through standardized protocols, incremental rollout, and automated observability checks.
This article explains how HuoLaLa’s stability team tackled the challenge of proving their work’s value by designing and implementing a comprehensive stability metric system from scratch, detailing the motivations, principles, step‑by‑step construction, data platform, cultural adoption, measurable results, and future plans.
This article explains why automation testing often lags behind product changes, outlines essential load‑testing concepts such as bottleneck analysis and capacity planning, and provides hands‑on guidance for using Wrk and GoReplay tools within an SRE‑driven operations workflow.
This article recounts a year‑long effort by a senior SRE engineer to redesign a private cloud platform, detailing the motivations, architectural choices, SSO and RBAC implementations, workflow automation, GitOps deployment, release engineering improvements, and the cultural shift toward metrics‑driven development.
This article outlines the evolution from manual operations to DevOps and SRE‑driven “big operations,” detailing system reliability and continuity practices, observability concepts, and the development of AIOps maturity standards, offering a comprehensive guide for building stable, efficient, and secure operational frameworks.
This article summarizes Zhang He’s Bilibili SRE talk on building a capacity‑management system that visualizes resource usage, reduces costs, improves stability, and leverages Kubernetes VPA, HPA, pooling, and quota management to support massive live‑stream events and rapid feature releases.
This article explains the fundamentals of Site Reliability Engineering, detailing how SRE combines development and operations to improve stability through metrics like MTBF and MTTR, the roles of SLI/SLO, the VALET selection method, and the practical use of error budgets for quantifying work and guiding alerts.
This article presents a comprehensive set of DevOps/SRE best practices—including hiding service provider resource addresses, installing only required dependencies, running only necessary services and ports, and using bastion hosts—to improve system security, reliability, and operational efficiency.
The article examines Vipshop's massive March 2023 outage caused by an IDC cooling failure, critiques superficial PPT‑driven reliability claims, and offers practical SRE insights on fault drills, true multi‑active architectures, and how ops teams can gain influence despite budget constraints.
This article explores how Tencent builds and practices its SRE quality operation system, focusing on On‑Call incident management, standardized channels, alert handling, data quality models, and the resulting improvements in reliability, MTTR reduction, and data‑driven decision making.
This article examines Tencent's end‑to‑end SRE quality‑operation framework built on Service Level Objectives (SLO) and On‑Call, detailing industry background, problem statements, SLO management, On‑Call benefits, product architecture, large‑scale deployment, and future plans for reliability engineering.
This article details Bilibili's SRE team's design and implementation of a high‑availability multi‑active architecture, covering zone types, same‑city and cross‑region deployments, traffic routing, cache consistency, message handling, governance, and practical lessons learned from real‑world incidents.
The article explains why striving for perfect service availability is unnecessary, outlines the cost of high reliability, shows how to measure availability and SLOs, discusses who should set SLOs, and highlights the importance of ROI when improving reliability.
This article traces the origins of Site Reliability Engineering from Margaret Hamilton’s pioneering work on the Apollo program, through Google’s formal SRE team creation, and highlights the key differences between SRE and traditional operations practices.
The SRE team diagnosed an in‑flight Wi‑Fi outage for the DeWu e‑commerce app by reproducing the problem, capturing packets with ping, traceroute and tcpdump, discovered a firewall rule misclassifying the domain as a download site, and resolved it through a vendor‑issued policy update, restoring connectivity on both ATG and SATCOM links.
The article explains that high metric cardinality in Prometheus causes long query times and timeouts, and demonstrates how using recording rules to pre‑compute aggregates dramatically reduces cardinality and latency, while recommending scrape interval tuning and metric design best practices to keep charts responsive.
This article presents a systematic overview of the emerging XOps ecosystem—including DevOps, BizDevOps, AIOps, FinOps, and SRE—detailing their relationships, maturity models, standards, and practical guidance for enterprises seeking to achieve efficient, secure, and data‑driven digital transformation.
This article explains why Site Reliability Engineering (SRE) and DevOps are both essential for modern software development, compares their objectives, outlines their complementary roles, and highlights the fundamental differences that help organizations achieve faster releases with higher reliability.
This article explains how Meta built SLICK, a centralized SLO/SLI platform that improves service reliability through discoverability, long‑term insights, integrated workflows, and scalable architecture, and shares real‑world examples and lessons learned from its deployment across thousands of services.
This article explains the fundamentals of distributed application monitoring, describes Huya's non‑intrusive data‑collection techniques using SDKs and plugins, outlines the design and correlation of observable metrics, and demonstrates practical results and troubleshooting scenarios for backend services.
This talk outlines the challenges of maintaining system stability in fast‑moving, cloud‑native financial services, describes a risk‑identification model, high‑fidelity fault simulation, and a comprehensive stability engineering platform, and shares future plans for automated, data‑driven risk mitigation.
At the 20th GOPS Global Operations Conference, China Mobile Jiangsu showcased how its capability platform leverages AI, big data, and blockchain to automate health scoring and intelligent inspection, dramatically improving container‑cloud operational efficiency and paving the way for smarter, SRE‑driven DevOps practices.
This article details Huolala's journey in establishing a comprehensive technical stability framework, covering organizational challenges, risk governance, incident response, cultural initiatives, and future automation to enhance system reliability at scale.
Since 2021, 得物 has built a FinOps practice that combines finance and DevOps to allocate, forecast, and optimize multi‑cloud spend, delivering over ¥100 million in annual savings through cost‑center platforms, container‑utilization improvements, custom PaaS services, automated analysis, and a cross‑functional virtual team governed by visibility, optimization, and governance KPIs.
This article explains the role of Site Reliability Engineering (SRE), its responsibilities, how it complements product development, the software lifecycle perspective, and practical approaches to ensure system stability through controllability, observability, and best‑practice implementation.
Bilibili's capacity management platform integrates foundational data, VPA/HPA scaling, quota control, and visual dashboards to streamline resource usage, cut costs, and boost stability, delivering event‑specific support such as for S12 that slashes release issues by 80% and online failures by 90%, while planning predictive scaling and risk control.
This guide outlines fifteen key DevOps and SRE tools for 2023—including monitoring, application platforms, chat‑ops, incident management, diagramming, and CI/CD solutions—explaining their core features, benefits, and how they help teams maintain reliable, observable, and automated software delivery pipelines.
This article explains the origins and meaning of software stability and stability testing, outlines key standards such as GB/T 16260 and industry definitions, and presents a comprehensive framework for stability quality assurance covering system elements, external disturbances, baseline setting, robust design, monitoring, and rapid incident response.
This article reviews how Tencent Health Code handled billions of daily scans by adopting cloud‑native architecture, comprehensive observability, capacity stress testing, chaos engineering, and disciplined change control to ensure high availability and resilience as pandemic demand waned.
The article details Bilibili’s Comet workflow engine—a low‑code, plugin‑extensible platform built since 2019 that uses visual DAG templates, graph‑based legality checks, and asynchronous execution to automate diverse business processes such as SRE automation, permission requests, and push‑task approvals, improving operational efficiency across mobile and web services.
Bilibili extended Kubernetes with a custom in‑place Vertical Pod Autoscaler framework—including generator, recommender, updater, and webhook controllers plus a management platform for strategy tuning, avoidance, analysis, and anomaly detection—reducing over‑provisioned resources across its ten‑thousand‑node private cloud and achieving up to 60 % CPU and 30 % memory savings.
DeWu’s container SRE team combines software‑engineered reliability with routine operations, using defined on‑call roles, SLO/SLA targets, progressive change management, capacity forecasting, four‑metric monitoring, MTTR/MTTF tracking, kernel‑parameter tuning, and namespace‑protected security policies to swiftly resolve incidents such as Redis latency spikes.
Site Reliability Engineering (SRE), introduced by Google in 2003, has become a cornerstone for ensuring the reliability and stability of large‑scale internet platforms, and Chinese experts now share home‑grown practices and a new book that distills two decades of SRE experience for building high‑availability applications.
The article reviews traditional availability metrics such as Success‑Ratio, Error‑Budget, MTTR/MTTF, SLA/SLO, and highlights their limitations, then introduces Google’s User‑Uptime and Windowed User‑Uptime metrics, explains their definitions, challenges, experimental results, and why they provide a more user‑centric view of service reliability.
The article explains Google’s Site Reliability Engineering (SRE) team, its mission to balance reliability and velocity through automation, the engagement model with development teams, funding principles, and a set of guiding principles that shape how SRE collaborates, scopes, and delivers value across services.
To cut the hundreds‑of‑thousands‑yuan monthly bill of a commercial HTTPDNS service, the team built a multi‑region, self‑hosted HTTPDNS platform, estimated to slash costs by up to 90%, then resolved unexpected TLS bandwidth waste by improving connection reuse, ultimately achieving over 80% savings and planning a hybrid‑cloud deployment.
This article explains how enterprises can design and implement a comprehensive observability platform—covering metrics, logs, tracing, fault response, post‑mortems, testing, capacity planning, and automation—to improve system reliability and user experience.
This article details Tencent CDN's challenges and solutions for business continuity, covering bandwidth and device resource constraints, massive request handling, fault‑management lifecycle, automation bottlenecks, and the implementation of AIOps, intelligent alerts, capacity planning, and root‑cause analysis to ensure reliable service.
This article chronicles Bilibili's five‑year evolution of Site Reliability Engineering, detailing the introduction of SRE culture, the construction of high‑availability and multi‑active architectures, capacity management with Kubernetes, VPA/HPA, incident case studies, and the ongoing transformation of SRE practices across the organization.
Bilibili’s S12 technical guarantee team coordinated dozens of engineering groups, performed resource estimation, built a shared resource pool, applied chaos engineering, high‑availability architecture, and systematic performance testing to ensure the League of Legends World Championship livestream remained stable and responsive under peak traffic.
This article details Tencent CDN's extensive business continuity challenges—including bandwidth, device resources, and massive request volumes—and explains how a fault‑management lifecycle, AIOps components, intelligent alerting, and automated capacity planning together enable resilient, automated operations.
The technical‑assurance team secured Bilibili’s high‑write live‑stream gift system by expanding capacity, isolating hot keys, refactoring pipelines, adding asynchronous writes, employing horizontal scaling and full‑link load testing, converting uncertain dependencies into graceful fallbacks, and deploying dual‑active, chaos‑engineered disaster‑resilience architecture aligned with business usage patterns.
This article explains how SRE teams should define Service Level Indicators, Objectives and Agreements, manage reliability, performance, saturation and observability, use proper metrics and tracing, handle error budgets, assess risks, and implement effective incident and project management to create robust, cloud‑native services.
This article explains why thorough, blameless incident postmortems are essential, outlines when to initiate them, describes the key components of an effective review, and offers practical steps to transform each outage into a continuous‑improvement opportunity for engineering teams.
To ensure “tea‑time” reliability for Bilibili’s 2022 S12 League of Legends championship, a cross‑functional technical‑assurance project introduced shared resource pools, CPUSET removal, multi‑instance HA architecture, adaptive throttling, chaos‑engineered fault injection, a new Golang gateway, extensive load testing, and coordinated on‑site duty, delivering uninterrupted live streaming without forced throttling.
The article reviews classic availability metrics such as Success‑Ratio, Incident‑Ratio, MTTR/MTTF, Error‑Budget, and SLA/SLO, then introduces User‑Uptime—a per‑user success time proportion that ignores long idle periods—and its windowed variant, showing how it complements existing indicators for more user‑centric reliability insight.
The article explains that SRE, introduced by Google, is not a single specialist but a collaborative mindset requiring product, development, testing, operations, and architecture skills, and argues that even small‑scale teams can achieve stability by embracing these principles despite common misconceptions.
The article clarifies the distinctions between platform engineering, DevOps, and SRE, explaining their origins, common misconceptions, challenges such as shadow operations and developer cognitive load, and how platform engineering builds on these practices to deliver self‑service internal developer platforms that improve productivity and reliability.
The article examines growing developer frustration with operational responsibilities, the perceived decline of DevOps, and how platform engineering and Site Reliability Engineering are emerging as new approaches to balance development speed with reliable operations in cloud‑native environments.
Clear responsibility boundaries are essential for effective DevOps and SRE workflows, preventing endless fire‑fighting, aligning software and organizational structures, and ensuring sustainable delivery across business stages. They also highlight how team size and expertise affect process clarity and when to choose DevOps‑first or SRE‑guided approaches.
This article analyzes the challenges of alert overload in large‑scale microservice environments and presents a systematic approach—including timeliness metrics, a maturity model, lifecycle tracking, feedback loops, downgrade mechanisms, and cross‑service aggregation—to improve alert effectiveness and reduce noise.
This article explains what a reliability culture is, why it matters, how to cultivate it through mission statements, early‑stage reliability testing, chaos‑engineering practices like GameDays and FireDrills, and how organizations can continuously learn from incidents to improve system availability and customer trust.
At the 2022 GOPS Global Operations Conference in Shenzhen, Bilibili’s infrastructure SRE lead Wu Anchuang unveiled the company’s comprehensive stability framework—detailing its SRE transformation, high‑availability architecture, active‑active disaster‑recovery, capacity planning, and event‑support strategies—marking the first public disclosure of these practices.
This article details the operational mindset, stability framework, and transformation journey of the Zuoyebang infrastructure team, covering service lifecycle management, standardization, cloud‑native architecture, multi‑active deployment, incident pre‑plan platforms, traffic scheduling, monitoring, capacity planning, and future directions for SRE service‑orientation.
The article clarifies common misconceptions about DevOps, explains that true development‑operations integration relies on dedicated ops teams, automation tools, standardized delivery artifacts, and unified permission management rather than developers performing ops tasks, and highlights Google SRE practices as a practical guide.
The article shares a senior architect's experience improving a legacy multi‑service web system's performance by 3000% by fixing a DB connection leak, adopting tail‑latency metrics, and investing in load testing, monitoring, logging, and dedicated SRE resources.
This article shares a real‑world case study of how a hidden database‑connection leak in a pod’s health‑check caused severe latency, and outlines four key lessons on performance metrics, testing, legacy system maintenance, and the critical impact of every line of code.
On July 13, 2021, Bilibili’s OpenResty‑based SLB suffered a CPU‑100% outage caused by a Lua _gcd function bug triggered when a service’s weight was set to the string “0”, leading to a multi‑hour incident that was resolved by rebuilding SLB clusters and disabling JIT compilation.
Bilibili adopted Google‑SRE SLO practices—selecting SLIs, defining availability and latency targets, grading services, and tracking error budgets—but encountered costly grading inconsistencies, hidden error detection, and inaccurate business‑level metrics, leading them to realize SLOs are chiefly valuable for early alerting rather than exhaustive reporting.
This article provides a comprehensive overview of Site Reliability Engineering, explaining its origins, core responsibilities across infrastructure, platform, and business layers, daily tasks such as deployment, on‑call duties, SLI/SLO management, incident post‑mortems, capacity planning, and user support, as well as career advice for aspiring SREs.
This postmortem details the July 13, 2021 Bilibili outage caused by a Lua‑induced CPU 100% bug in the OpenResty‑based SLB, describing the incident timeline, root‑cause analysis, mitigation steps, and the subsequent technical and process improvements to enhance reliability and multi‑active deployment.
This post‑mortem details the July 2021 Bilibili outage caused by a Lua bug in the OpenResty‑based SLB, describing the timeline, root‑cause analysis, mitigation steps, and the technical and organizational improvements implemented to prevent similar incidents.
In July 2021 a sudden CPU‑100% spike in Bilibili's OpenResty‑based SLB caused widespread service outages, prompting an emergency response that rebuilt load‑balancer clusters, traced a Lua _gcd function bug triggered by a zero weight string, and led to extensive operational and architectural improvements.
On July 13 2021 Bilibili’s L7 SLB crashed when a recent Lua deployment set a balancer weight to the string “0”, producing a NaN value that triggered an infinite loop and 100 % CPU, prompting emergency restarts, a fresh cluster rollout, and long‑term safeguards such as automated provisioning, stricter Lua validation, and enhanced multi‑active disaster‑recovery processes.
The article outlines the nine core competencies—network expertise, Linux/Unix knowledge, cloud computing, CI/CD pipelines, QA automation, security engineering, DevOps, incident management, and post‑incident review—that enable SREs to ensure the availability, performance, and reliability of complex distributed systems.
A seasoned operations professional shares personal experiences and hard‑earned insights on why traditional monitoring often becomes ineffective, how over‑automation and noisy dashboards hurt teams, and what a capability‑focused, user‑centric approach to observability should look like.
This article details Huya's SRE-driven architecture that combines center‑edge deployment, high containerization, AI‑powered elasticity, fault avoidance, and fast recovery mechanisms to achieve deterministic, highly available live‑streaming services.
This article explains the motivations behind Site Reliability Engineering (SRE), outlines its strategic goals, defines key concepts such as SLI, SLO, SLA and error budget, introduces the four golden metrics for monitoring distributed systems, and provides practical guidance on building, operating, and continuously improving an SRE practice.
This article describes how the Haodafu offline push service grew from a simple PHP notification tool into a robust, highly‑available micro‑service platform by redesigning architecture, adopting vendor push channels, adding message‑queue reliability, implementing comprehensive monitoring, observability, and a fault‑diagnosis platform to ensure delivery rates and operational stability.
Industry experts from Zhejiang Mobile, Bilibili, and Xiaomi discuss how SRE has evolved in the cloud‑native era, sharing concrete frameworks, observability practices, and cost‑focused platforms while emphasizing stability, metrics, on‑call processes, and the need to adapt Google’s model to real‑world product and operational contexts.
This article details Bilibili's SRE approach to large‑scale live events, covering background, activity scenarios, resource planning, performance testing, chaos‑engineering drills, technical safeguards such as DCDN, SLB, WAF, PaaS, cache and DB, pre‑plan capabilities, post‑mortem analysis, and future outlook, illustrating how systematic capacity management and automated resilience practices enable stable operation for events with tens of millions of concurrent users.
This article recounts a three‑day image‑upload outage in a mini‑program, analyzes the multi‑layer causes, and shows how combining Metrics‑Driven Development with SRE and a custom observability platform dramatically reduces diagnosis time and improves reliability.
The article examines why SRE teams experience overload due to high incident response demands, analyzes contributing factors such as production issues, alert volume, and manual processes, and proposes comprehensive mitigation steps including better testing, load management, and proactive error detection to reduce on‑call burden.
After a three‑day, ten‑person investigation into a mini‑program image‑upload failure, we designed and implemented an end‑to‑end observability platform using MDD and SRE principles, defining SLI/SLO, instrumenting client, network, gateway and backend layers, and visualizing metrics with Grafana, ClickHouse and Prometheus.
This article explains Site Reliability Engineering (SRE), its core concepts such as SLI, SLO, SLA, error budgets, risk analysis, the four golden metrics, and practical steps for developing, piloting, and operating reliable services with monitoring, automation, and post‑mortem practices.
Bilibili’s SRE team, confronting rapid growth and complex systems, built a systematic stability operation that includes emergency response, incident handling, on‑call scheduling, and an Event Operations Center platform, using metrics like MTTR, MTTI and AI‑assisted automation to reduce downtime and improve reliability.
The article explains how cascading failures arise from positive feedback loops in distributed systems, illustrates real‑world incidents such as the 2015 DynamoDB outage, outlines anti‑patterns like unlimited retries and unchecked load, and presents practical mitigation techniques including load‑shedding, circuit breakers, exponential back‑off, and controlled replication to improve system resilience.
Bilibili’s SRE team combines stability theory, detailed fault‑stage and operational metrics, and a unified emergency‑response platform—including on‑call scheduling, fault‑command incident commanders, automated fault portraits, and rapid post‑mortems—to transform frequent incidents into data‑driven, collaborative recoveries and lay groundwork for AI‑assisted self‑healing.
This article explains what Site Reliability Engineering (SRE) is, why it was created, the challenges in hiring SREs, and breaks the role into three layers—Infrastructure, Platform, and Business—detailing their duties, deployment processes, on‑call practices, SLI/SLO management, incident post‑mortems, capacity planning, user support, and career advice.
This article explains what SRE is, why it was created, how its responsibilities differ across companies, and breaks the work into Infrastructure, Platform, and Business SRE while covering deployment, on‑call, SLI/SLO, incident post‑mortems, capacity planning, user support, and career advice.
This article explains Site Reliability Engineering (SRE), covering its origins, core responsibilities, key concepts such as SLI/SLO/SLA and error budgets, the four golden monitoring metrics, risk analysis, and practical guidance on building reliable services using tools like Prometheus and Grafana.
The article explains the rising importance of observability in modern operations, defines its control‑theory roots, breaks it down into metrics, traces and logs, and argues that successful implementation requires three pillars—SRE practices, AIOps algorithms, and deep business‑architecture knowledge—together with well‑designed SLOs and critical‑path mapping.
This article presents a hands‑on SRE framework covering the full product lifecycle—code development, resource planning, deployment, operational reliability, and decommissioning—derived from real‑world practices at Xiaomi and Sina to help teams manage massive internet services efficiently and cost‑effectively.
The article describes how a three‑day, cross‑team investigation of a mini‑program image‑upload failure led to the design and implementation of an SRE‑driven, metrics‑driven observability platform that quantifies SLIs, automates tracing, and provides dashboards for real‑time and long‑term analysis, ultimately reducing MTTR.
Observability is more than just combining metrics, traces, and logs; successful implementation requires the disciplined SRE methodology, AI‑driven AIOps capabilities, and a deep understanding of business architecture to define critical paths and layered SLOs for real‑world systems.
Drawing from years of ops experience, the author recounts the decline of traditional monitoring, the rise of automated dashboards, the challenges of AIOps and observability, and proposes a shift toward data‑driven, business‑focused capability building to make alerts truly useful.
This article shares practical lessons from optimizing the performance of legacy web applications, covering how a hidden DB‑connection leak caused severe latency, why average latency is misleading, and the essential tools, testing, and team practices needed to keep services fast and reliable.
This article explains the SRE role originated by Google, outlines its core responsibilities such as automation, observability, incident response, testing, capacity planning, and SLI/SLO/SLA management, and highlights the skills and cultural practices needed for reliable service operations.
After a sudden traffic surge exposed severe latency in a 15‑year‑old multi‑service web system, the team identified a DB connection leak in a health‑check probe, and through load testing, monitoring, logging, and dedicated SRE effort, they derived key operational lessons on performance optimization and maintenance of legacy applications.