This article shares Anthropic’s SRE engineer insights on 23 actionable practices—from schema migration and Karpenter node management to OpenTelemetry adoption, Helm chart storage, and Terraform versus CloudFormation—offering concrete recommendations for building reliable, cost‑effective AI and cloud‑native platforms.
The article introduces China’s System Operation Maturity Model (SOMM), detailing its three pillars—AIOps, SRE, and FinOps—along with the underlying standards, assessment results, and how enterprises leverage these frameworks to achieve smarter, more reliable, and cost‑effective IT operations.
Chaos engineering is a discipline that deliberately injects faults into distributed systems to test and improve resilience, tracing its evolution from Netflix's Chaos Monkey to modern platforms, outlining its operational workflow, benefits, and core principles for reliable system design.
This guide explains the four SRE golden signals—Latency, Traffic, Errors, and Saturation—covers their definitions, how to measure them with Prometheus in Node.js, compares them to RED and USE frameworks, and provides concrete alerting rules for reliable service monitoring.
Baidu’s SRE digital‑immune system, evolved into an AI‑powered intelligent immunity platform, quantifies and mitigates risk across thousands of services by integrating data‑driven monitoring, rule‑based detection, and large‑model GraphRAG knowledge mining, cutting degradation cases by ~40% and shifting operations from reactive troubleshooting to proactive, data‑centric quality assurance.
This article explains why modern SRE teams need a digital immune system, describes Baidu’s data‑driven approach to improve system resilience, outlines the three‑phase evolution from digital transformation to AI‑enhanced risk mining, and shares concrete results and future directions for sustainable operations.
This article reviews nine practical container‑monitoring solutions—from Last9 and Prometheus to Dynatrace and Elastic Observability—detailing their key features, pricing, and why developers prefer them, and then offers comprehensive best‑practice guidance for metrics, tagging, alerts, and advanced observability strategies in Kubernetes‑driven cloud‑native deployments.
This guide explains how SRE teams should collaborate early in the software development lifecycle to define Service Level Indicators (SLIs), set realistic Service Level Objectives (SLOs) and Service Level Agreements (SLAs), and integrate observability signals, error budgeting, risk management, and incident handling into reliable operations.
This article outlines fifteen common operational and management mistakes—such as frequent incidents, excessive new hires, lack of automation, and missing rollback plans—that can trigger system outages, and offers guidance on how teams can strengthen testing, processes, and team capabilities to prevent downtime.
Operations leaders and CTOs often confront the question of the true value of their teams, and this article explores who asks it, why it matters, typical challenges, and practical ways to define and protect the operational role through unified platforms, processes, and strategic collaboration with development.
Chaos engineering, introduced by Netflix in 2011 to proactively inject failures and test system resilience, has evolved over the past decade into a widely adopted practice integrated with SRE, automation, AI, and Kubernetes, offering best‑practice guidelines and future trends for improving distributed system reliability.
This article compiles diverse Zhihu comments on the reality of 24 × 7 on‑call duties, contrasting exaggerated myths with practical team‑based solutions, global shift models, backup strategies, and actionable tips for improving operations without sacrificing personal life.
This article explores how open‑source large language models like DeepSeek empower SRE teams to shift from reactive firefighting to proactive, predictive operations, detailing technical principles, real‑world case studies, essential skill sets, and future trends that reshape the operations landscape.
This article shares a step‑by‑step account of taking over a content‑risk stability role in early 2024, defining system stability, diagnosing recurring issues, and implementing a three‑phase framework—pre‑emptive reduction, impact mitigation, and post‑incident improvement—to boost success rates, cut incident response time, and achieve a modular architecture.
This article explains how SRE teams should collaboratively define Service Level Indicators, Objectives, and Agreements, and then cover reliability, performance, observability signals, error budgeting, risk management, incident handling, and the engineering work needed to build robust cloud‑native platforms.
The article summarizes Lai Kunchi's presentation at the 24th GOPS Global Operations Conference, covering the current state and challenges of database development, Guangdong Mobile's database operation system, and future directions for managing heterogeneous databases in evolving business architectures.
This article explores the hierarchy of operations roles, outlines five career stages from entry‑level to AI‑driven expert, and offers practical advice on building foundations, automation, high‑availability design, and embracing emerging technologies.
Many companies deploy fragmented monitoring and observability tools yet still struggle to pinpoint incidents; this article analyzes the root causes—under‑utilized tools and scenario‑agnostic data—and offers practical steps to organize metrics, build layered insights, and improve fault‑resolution efficiency.
At the 2024 GOPS Global Operations Conference in Shanghai, Alipay’s monitoring lead Tang Liang presented the challenges, architecture, risk‑prevention practices, and implementation details of the company’s full‑ecosystem availability monitoring system, highlighting its role in DevOps, SRE, and AIOps initiatives.
This article outlines comprehensive stability assurance mechanisms—including standards, process workflows, the distinction between developers and SREs, personal responsibilities, and practical construction directions—to guide teams in building resilient, high‑availability systems through proactive, daily, and incident‑response practices.
This article outlines a comprehensive stability framework for SRE teams, detailing essential mechanisms such as review processes, coding standards, incident management, on‑call responsibilities, and daily operational practices, while also highlighting the cultural shift needed to achieve reliable, high‑availability systems.
This article explores why influence matters for Site Reliability Engineers, outlines the challenges they face in gaining recognition, and provides practical strategies—enhancing technical expertise, improving communication, quantifying achievements, and sharing knowledge—to elevate their impact within organizations.
The article recaps Li Jingkang’s presentation at the 2024 GOPS Global Operations Conference, detailing the background, principles, design, and real‑world implementation of Qunar’s pre‑release platform, and outlines its future direction within DevOps, SRE, AIOps, and cloud‑native practices.
This article examines how the rapid rise of AI reshapes Site Reliability Engineering by enhancing monitoring, automating operations, improving fault diagnosis, and presenting new challenges such as data security and model explainability, ultimately driving more efficient and reliable system management.
This article compiles real‑world operations incidents—from accidental database deletions and faulty deployments to hidden data tampering and network device failures—highlighting how quick diagnosis, preventive maintenance, and SRE practices can mitigate impact on users, reputation, and revenue.
This comprehensive guide explains how to design, build, and operate high‑availability systems by covering availability metrics, fault‑tolerance strategies, capacity planning, code and data layer architecture, automated testing, monitoring, and clear role responsibilities to ensure services stay reliable and resilient under load.
Reviewing the major service disruptions—from Alibaba Cloud to OpenAI—this article extracts key SRE lessons such as early disaster‑recovery planning, regular backups, load balancing, real‑time monitoring, performance tuning, and capacity planning, urging enterprises to adopt resilient practices for a more stable future.
The article explains how early detection and preventive measures—such as comprehensive monitoring, rate limiting, chaos testing, and proper SLOs—are essential for maintaining system stability and avoiding larger incidents, drawing on SRE principles and the incident triangle theory.
The article outlines China’s 2024‑2027 information standard action plan, the rollout of ITU DevOps and AIOps assessments, and showcases dozens of telecom projects that achieved significant improvements in delivery speed, reliability, automation and observability through standardized DevOps, SRE and AI‑ops practices.
This article explains how system stability depends on architecture and code details, defines SLA and the “nines” metric, outlines Google’s SRE hierarchy, and provides practical governance steps—including development and release processes, high‑availability design, capacity planning, monitoring, incident response, and team culture—to achieve reliable, high‑availability services.
The recent Alipay outage during Double‑11 revealed a partial failure in its system message database, prompting users to experience payment errors, duplicate charges, and delayed withdrawals, while the company’s response highlighted the importance of comprehensive backup, redundancy, disaster‑recovery planning, monitoring, and security measures to ensure service continuity.
Amid a surge of high‑profile outages, the CAICT introduces a comprehensive SRE framework that addresses large‑scale, high‑frequency changes, complex tech stacks, and massive traffic, outlining development and operational reliability practices, maturity levels, and actionable guidelines to enhance system stability.
Understanding system health is essential for reliable services, and this guide explains how to use powerful monitoring tools to collect, visualize, and alert on the four golden signals—latency, traffic, errors, and saturation—across servers, applications, and external dependencies, helping teams detect and resolve issues efficiently.
The article explores the growing importance of system reliability in China, the national regulations driving SRE adoption, Inner Mongolia Mobile’s successful Level‑3 SRE assessment at the 2024 GOPS conference, and insights from Deputy GM Zhang Yongtao on practices, challenges, and future plans.
This article reveals three common pitfalls in daily incident postmortems—overlooking minor failures, confusing root causes with triggers, and weak improvement actions—and offers practical steps like the 5 Whys method and essential corrective measures to truly reduce online outages.
This article explains how Bilibili designed and implemented an Emergency Response Center (ERC) to manage the full fault lifecycle—detection, response, delimitation, localization, mitigation and recovery—using alert rules, automated recall, integrated customer feedback, clear role assignments, mobile support, and data‑driven post‑mortems, ultimately reducing MTTR and improving service reliability.
This article explains the concept of observability, contrasts it with traditional monitoring, outlines its benefits for system stability, reliability and performance, and provides practical guidance on building a full‑stack observability platform using logs, metrics, tracing and modern cloud‑native tools.
This article recounts how a solo developer built a Django‑based Docker image signing service to meet PCI DSS requirements, faced two severe incidents—including a 17.5‑hour P0 outage caused by concurrency limits and a misconfigured Rekor service—and shares the operational lessons learned for reliable SRE practice.
Bilibili’s Emergency Response Center (ERC) unifies incident detection, rapid response, precise scoping, and coordinated recovery through multi‑dimensional alerts, automated collaboration, standardized updates, and post‑mortem analysis, targeting 1‑minute detection, 3‑minute response, 5‑minute scoping, and 10‑minute recovery, which has cut MTTR, achieved over 80% automatic recall accuracy, and met more than 60% of its 1‑3‑5‑10 performance goals.
This article explains how iLogtail, Alibaba Cloud's high‑performance observability agent, tackles reliability challenges at million‑scale deployments through a comprehensive SRE workflow that spans design, development, testing, gray‑release, operations, and continuous customer support, all while leveraging cloud‑native tools and automation.
This article details how Alibaba Cloud's iLogtail agent, serving tens of thousands of hosts and containers, overcomes unique stability challenges through a comprehensive SRE approach that spans design, development, testing, gray‑release, operations, and customer‑support, ultimately boosting reliability and reducing incident rates.
This article explains what MTTR is, why it matters for system stability, and provides a step‑by‑step framework—including monitoring, alert tuning, rapid mitigation, clear role assignments, and post‑mortem practices—to dramatically shorten repair times and improve overall reliability.
The article outlines Google’s release engineering principles, roles, and processes—including self‑service, frequent high‑speed releases, sealed builds, policy enforcement, configuration management, and the Rapid system—to illustrate how automated, reliable software delivery is achieved at scale.
The article outlines SRE‑based incident mitigation and recovery practices, covering urgent mitigations, impact reduction, key metrics such as TTD, TTR, TBF, and detailed strategies for shortening detection and repair times, preventing fatigue, improving observability, and designing resilient systems.
This article explains why regular disaster role‑playing, systematic testing, and focused responder preparation are essential for building robust incident response capabilities and reducing operational risk in production environments.
This guide explains how to use the four golden signals—latency, traffic, errors, and saturation—to design effective monitoring across servers, services, and external dependencies, helping teams detect issues early and maintain reliable, high‑performance systems.
This article explores how service availability is quantified, explains the impact of MTBF and MTTR on reliability, and presents concrete operational practices—including redundancy, traffic control, and change‑management techniques—to move systems from basic uptime to true high‑availability levels.
This article presents a detailed curriculum covering fundamental infrastructure, cluster architecture, automation, log collection, Linux system administration, containerization, monitoring, security, and related DevOps tools across multiple phases and daily modules for comprehensive SRE training.
This article demystifies Site Reliability Engineering (SRE) by explaining its origins, core responsibilities, essential skill sets, and key practices such as observability, incident response, testing, capacity planning, automation, user support, on‑call duties, and the definition of SLI/SLO/SLA, providing a comprehensive guide for modern operations teams.
The article outlines a three‑stage evolution of operations—from rapid incident response to proactive fault‑injection—while offering practical guidance on improving availability, visualizing changes, and aligning technical metrics with business value to elevate the role of operations engineers.
Drawing on twenty years of Google’s SRE experience, this article shares eleven practical lessons—from proportional incident mitigation and pre‑tested recovery mechanisms to canary releases, disaster‑resilience testing, and frequent deployments—aimed at improving reliability and operational efficiency.
This interview recounts Hervé Le Meur’s path from a B2B marketing consultant to a Jenkins infrastructure SRE, detailing his work with Jenkins X, Kubernetes, CI/CD pipelines, and the lessons he shares for newcomers to open‑source contributions.
The article examines why many organizations avoid upgrading databases despite end‑of‑life warnings, highlighting cultural resistance, skill gaps, security and performance risks, and offers practical strategies for planning and executing safe database upgrades.
This article explains the definitions of stability and high availability, distinguishes their relationship, outlines key performance indicators, and provides a comprehensive framework—including fault prevention, detection, and recovery, as well as design, coding, testing, monitoring, and emergency response practices—to help teams build reliable, highly available systems.
At the 2024 GOPS Global Operations Conference, Bilibili senior R&D engineer Gu Lintao will present BCM—Bilibili’s Chaos Engineering Platform—showcasing how its design and capabilities let developers, testers, and SREs safely inject faults, uncover hidden architectural risks, and improve service stability through real‑world drills and systematic reliability engineering.
This article explains what Site Reliability Engineering (SRE) is, outlines the three main layers of SRE work—Infrastructure, Platform, and Business—covers hiring challenges, daily duties such as deployment, on‑call, SLI/SLO management, capacity planning, user support, and offers practical interview and career advice.
Amid economic growth and frequent continuity incidents, major internet firms are redefining SRE roles, emphasizing cost reduction, system resilience, risk management, AI‑driven operations, and close collaboration with development teams, as revealed by a detailed analysis of recent job postings from Ant Group, Alibaba, ByteDance and others.
This article explains why Site Reliability Engineering (SRE) emerged, outlines its core responsibilities, required skill set, and how SRE teams use SLOs, monitoring, and scenario drills to improve system reliability, performance, and observability in complex production environments.
This article outlines the rising frequency of system outages, explains the key characteristics and challenges of modern large‑scale distributed systems, introduces China’s CAICT SRE framework and its two‑part reliability model, showcases a successful SRE case, and announces the 2024 SRE maturity assessment program.
This article compiles several Zhihu answers that explain the various operations (运维) roles—SRE, network, system, DBA and development—describe their responsibilities, challenges, and career prospects, and answer whether the work is boring and if it’s worth pursuing.
This article explains the role of an operations (运维) engineer, covering daily responsibilities, essential knowledge such as Linux and networking, common monitoring tools, and emerging career paths like DevOps, AIOps, and SRE, helping newcomers understand how to start and grow in the field.
The article examines common operational pain points such as cumbersome release processes, lack of standardization, and weak security controls, then explores how AI‑powered SRE tools and automation can address these challenges and guide teams toward more efficient, standardized, and resilient operations.
The article examines common operational pain points such as cumbersome release processes, lack of standardization, and weak security controls, then presents community suggestions emphasizing automation, standardized workflows, CMDB, and AI‑driven SRE tools, concluding that clear direction outweighs perfect execution.
A collection of Zhihu contributors share practical experiences and opinions on whether a single operations engineer can handle the massive scale of 40,000 servers, covering workload, automation gaps, budgeting, hardware failure rates, and the necessity of team‑based high‑availability practices.
The article examines the wide range of opportunities for operations engineers, highlighting the low‑entry, high‑potential nature of the field, and offers practical advice on city selection, education, in‑demand technologies, soft‑skill development, and health to help professionals climb from basic desktop support to high‑paying DevOps or SRE roles.
The 21st GOPS Global Operations Conference in Shanghai featured Shenwan Hongyuan Securities' Yusi Song presenting an in‑depth look at automated operations, covering achievements, experience summaries, and future plans, with slide images and a downloadable PPT for attendees.
The article examines why 2023 saw a surge in major online outages, linking layoffs and cost‑cutting to lost expertise, and explores the entropy and Murphy laws that make perpetual high availability impossible without continuous, systematic investment and cultural change.
At the 21st GOPS Global Operations Conference, New Oriental's senior operations manager Qi Chen detailed the demand, technical, and focus pressures that drove a phased, full‑process observability standardization, leveraging OpenTelemetry, Telegraf, Loki and CMDB tagging to achieve cost reduction and higher stability.
In the context of digital transformation, six leading Chinese telecom operators applied the CAICT DevOps Capability Maturity Model to evaluate dozens of projects, achieving significant improvements in continuous delivery, technical operations, security, and AIOps, providing valuable references for the industry.
This article outlines a comprehensive, step‑by‑step approach for SRE incident commanders, covering fault perception, grading, team organization, remediation tactics, transparent communication, and post‑mortem practices to efficiently resolve service disruptions.
The article analyses why truly never‑failing systems cannot exist—citing entropy and Murphy’s laws—examines the organizational and technical obstacles to continuous high availability, and offers practical cultural and engineering practices such as testing, code review, monitoring, and regular system health checks to mitigate risk.
Building continuously available systems in 2023 is hampered by entropy‑driven technical debt and Murphy’s Law failures, and the “Divine Doctor Paradox” shows that successful availability work goes unnoticed while blame follows any outage, making cultural commitment—not just technology—the essential solution.
ChangePilot is Bilibili’s unified change‑management platform that standardizes change definition, lifecycle, and risk governance through a platform‑scenario model and five control levels (G0‑G4), offering built‑in checks, searchable records, subscription alerts, intelligent correlation, and emergency channels to boost production stability while maintaining operational efficiency.
This article examines how G Bank builds an application monitoring system based on ITIL and Google SRE principles, identifies its shortcomings, and evolves the platform into a visualized operations solution that aligns technical and business perspectives for faster incident resolution and improved customer experience.
AlterShield is an open‑source change‑risk prevention solution originally built by Ant Group that provides lifecycle‑aware change defense, cloud‑native operator integration, KDE‑based anomaly detection, and extensible plug‑in frameworks, with detailed module descriptions, recent v1.0 releases, and a roadmap for advanced monitoring and noise‑reduction capabilities.
Bilibili revamped its alert monitoring platform to meet rapid growth, focusing on effectiveness, timeliness, and coverage; it introduced a closed‑loop design and governance that cut weekly alerts by 90%, built a knowledge‑graph root‑cause system achieving 87.9% accuracy with sub‑minute latency, and integrated AIOps for ongoing refinement.
This article examines the wide salary range in the operations field, explains why high‑level DevOps/SRE positions command premium pay, and offers practical advice—such as relocating to major cities, obtaining a bachelor's degree, mastering in‑demand technologies, and developing soft skills—to help operations engineers advance their careers.
Drawing on twenty years of Google SRE experience, this article outlines eleven practical lessons—from scaling mitigation to disaster‑resilience testing—that help teams design, operate, and evolve reliable large‑scale services.
On November 25, Dewu Technology hosted an SRE Stability Engineering salon in Hangzhou where experts from Alibaba, Tencent, Ant Group and Dewu shared practical insights on C‑end link reliability, Alibaba’s system stability operations, Tencent Game’s cloud‑native SRE practices, and Ant Group’s chaos engineering, concluding with a Q&A and resource distribution.
After a series of change‑induced outages in early 2023, Bilibili instituted a comprehensive change‑management framework—including a preventive change platform, a central control system, quality and monitoring tools, strict gray‑release policies, observability checks, and rapid rollback mechanisms—to dramatically cut emergency incidents and embed a reliability‑first culture.
The article details how Beijing Mobile’s SRE Smart Operations team applied SRE principles, automation, and cloud‑native tools to transform traditional DevOps into a reliable, scalable operation, highlighting their fault‑prevention, monitoring, incident response, and continuous improvement practices that earned them the 2023 IT Technology Leadership award.
This article describes how Huya built a unified metadata platform to break data silos across its numerous operations systems, enabling standardized data ingestion, association, visualization and analysis that improve resource governance, root‑cause diagnosis, and overall cost‑control for SRE teams.
This article explains the concept and importance of MTTR (Mean Time To Repair), shows how to calculate it, and provides a comprehensive set of monitoring, alerting, rapid mitigation, tool‑assisted analysis, and team coordination techniques to significantly shorten incident resolution time and improve system reliability.
This article describes how Huya built a unified metadata platform to break data silos across its SRE systems, enabling standardized data ingestion, correlation, and analysis that improve resource governance, root‑cause diagnosis, and overall cost‑efficiency for large‑scale live streaming services.
This article explains Site Reliability Engineering (SRE) as a collaborative methodology, outlines its stability goals measured by MTBF and MTTR, details how SLI/SLO and the VALET selection guide fault detection, and shows how error budgets quantify reliability work and drive precise alerting.
Beijing Mobile’s Order Center project passed the CAICT’s Level‑3 System Reliability and Continuity Engineering assessment, showcasing how SRE practices, cultural shifts, and tool automation boosted system stability, reduced incidents by 77%, cut recovery time by 54%, and set a benchmark for large‑scale IT operations in China’s telecom sector.
The Industrial and Commercial Bank of China software development center created an SRE panoramic monitoring view system that unifies data channels, standardizes metrics, offers multi‑dimensional dashboards, and introduces an intelligent Ops Assistant, dramatically improving fault detection, response speed, and cross‑team operational efficiency.
The article details Beijing Mobile's successful System Reliability Engineering (SRE) assessment at the 2023 GOPS Global Operations Conference, highlighting the company's SRE transformation, the benefits achieved, challenges faced, and future plans for scaling reliable IT operations across the enterprise.
China Agricultural Bank’s two flagship projects—Distributed Core Open System Control and Online Payment Platform—successfully passed the CAICT DevOps Technical Operations Level‑2 assessment, showcasing advanced domestic maturity, detailed implementation practices, and the broader impact of standardized DevOps on banking digital transformation.
The article explains platform engineering as the evolution of DevOps into a productized internal infrastructure function, detailing its definition, target users, responsibilities, organizational placement, differences from DevOps and SRE, industry trends, implementation practices, and criteria for evaluating the need for a dedicated platform team.
This article shares KuJiaLe's experience tackling stability challenges caused by rapid user growth and system complexity, detailing their organizational, process, cultural, and technical approaches—including goal setting, a stability committee, monitoring, incident response, change control, and regular drills—to achieve measurable improvements in reliability and performance.
The article explains how Home Depot (THD) adopted the VALET pattern language—Volume, Availability, Latency, Error, and Ticket—to unify service‑level objectives, automate data collection, build dashboards, and improve SRE practices across its massive retail and e‑commerce infrastructure.
This article explains the essential fault metrics MTTR, MTBF, and MTTF, their definitions, calculations, and practical importance for SRE and operations teams to improve system availability, guide maintenance strategies, and make data‑driven reliability decisions.
This article explores the four pillars of technical operations—Dao (philosophy), Fa (methodology), Shu (techniques), and Qi (tools)—detailing concepts from business continuity to SRE, FinOps, GitOps, CMDB design, and the underlying logic of modern ops platforms.
The article explores why building system stability is crucial by linking entropy theory to software reliability, introduces the availability formula, discusses common pitfalls and industry practices, and proposes a three‑stage governance framework—prevention, mitigation, and post‑mortem—to systematically improve operational resilience.
The article explains how Google measures and improves software quality before release by separating development and operations responsibilities, using monorepo and trunk‑based development, daily release candidates, automated testing, performance benchmarks, and a comprehensive Project Health (pH) metric system that balances speed, reliability, and quality.
This guide explains how SRE teams should define service level indicators, objectives, and agreements, design reliable and observable architectures, manage error budgets, assess risks, handle incidents, and integrate development practices to improve system stability and performance.
Dewu’s capital‑loss prevention framework embeds risk assessment and technical safeguards—such as idempotency, distributed consistency, and active‑active multi‑region design—into architecture, organizes three defensive lines (development, QA, SRE), and employs real‑time, near‑real‑time, and offline verification plus regular drills, while advancing automated analysis and intelligent scaling.