After a traffic surge exposed severe latency in a 15-year-old multi-service web platform, we used monitoring to discover a DB-connection leak caused by a liveness probe, corrected it, and distilled four practical lessons on latency metrics, tooling, legacy maintenance, and code vigilance.
The article shares a real‑world case study of a legacy multi‑service web platform where traffic spikes exposed DB connection leaks, leading to a 90% response‑time bottleneck, and outlines four key takeaways about tail‑latency metrics, investing in tools and people, actively maintaining legacy systems, and treating every line of code as critical for performance.
Huawei Cloud’s internal “blue‑team” launched over twenty coordinated attacks around Chinese New Year, but the company’s SRE “red‑team” and a dedicated 24/7 “special forces” unit detected, isolated, and resolved incidents within minutes, keeping failure rates below 0.01% and demonstrating advanced cloud operations and security practices.
This article presents a comprehensive study of continuous operations, defining its meaning, outlining the three key stages of continuous deployment, operation, and feedback, reviewing ITIL and DevOps practices, and sharing real-world case studies from major tech companies to illustrate effective implementation.
Over two years, we built a monitoring system covering 200+ services and 700+ instances, evolving from ad‑hoc Nginx logs to a Prometheus‑based observability platform with unified dashboards, automated alerts, and lessons on metric selection, alert fatigue, and fault isolation.
Phoenix is Meituan’s client‑side CDN disaster‑recovery system that uses a Webpack‑based SDK, dynamic calculation service, and monitoring platform to automatically detect load failures, switch domains, isolate problems, and continuously hot‑standby resources, boosting resource success rates from 99.7 % to 99.9 % across hundreds of projects.
This article details Industrial and Commercial Bank of China's comprehensive MySQL governance framework, covering risk identification, prevention strategies, a four‑step methodology, automated quality gates, production‑level monitoring, SRE management, and future visions for rapid incident detection and self‑healing.
This article presents a concise collection of ten Chinese-language resources that summarize the 2021 DORA DevOps report, the importance of consistency in R&D, fundamental efficiency principles, Microsoft’s testing shift, Google’s release and productivity metrics, and SRE health measurements, offering valuable insights for modern software engineering teams.
This article explains the origins of Site Reliability Engineering, breaks down its three main layers—Infrastructure, Platform, and Business SRE—covers day‑one and day‑2 deployment, on‑call processes, SLI/SLO design, post‑mortems, capacity planning, user support, and offers practical advice for aspiring SREs.
This article explains the role of Site Reliability Engineering (SRE), its origins at Google, the challenges of hiring, the three-layer model of infrastructure, platform, and business SRE, and provides detailed responsibilities, on‑call practices, SLI/SLO management, capacity planning, and career advice for aspiring SREs.
This article explains the fundamentals of Site Reliability Engineering, outlines a complete SRE workflow from prevention to post‑mortem, details key availability metrics and golden indicators, examines how technical debt arises and can be mitigated, and describes the tooling and practices needed to keep large‑scale services healthy.
This talk recounts how the Good Doctor platform tackled severe online incidents by launching the DOA project, then a service risk governance initiative that identifies, quantifies, and mitigates latency‑related risks through metrics‑driven development, dependency analysis, middleware reliability, and a dedicated risk‑management platform.
This article introduces the fundamentals of Site Reliability Engineering (SRE), explains how to measure service stability with metrics like MTTR, MTBF, and availability, outlines the SRE workflow from prevention to post‑mortem, and discusses how to identify and reduce technical debt to improve system health.
The 2021 DORA DevOps Report reveals that elite teams outperform low‑performing teams by adopting SRE principles, high‑quality documentation, integrated security, modern technical practices such as loose coupling, continuous testing, CI/CD, and a performance‑driven culture that fosters belonging and inclusion.
The 2021 DORA DevOps Report, based on a seven‑year study of over 32,000 professionals, reveals how elite software delivery and technical‑operations practices—such as reliability goals, secure supply‑chain integration, high‑quality documentation, cloud adoption, and positive team culture—drive superior organizational performance and provides data‑driven guidance for improvement.
The 2021 DORA Accelerate State of DevOps report, based on responses from over 32,000 professionals, reveals new performance metrics, the impact of SRE and security supply‑chain practices, cultural factors affecting burnout, and how cloud adoption continues to drive higher software delivery and organizational performance.
Google’s 2021 DORA Accelerate State of DevOps report, based on over 32,000 professionals, reveals that elite teams dramatically outperform low‑performing teams across deployment frequency, lead time, recovery time and failure rates, while highlighting new reliability metrics, the importance of team culture, SRE, cloud adoption, secure software supply chains and documentation.
The 2021 Accelerate State of DevOps report by Google DORA, based on over 32,000 professionals, reveals that elite teams dramatically outperform low‑performing teams across four classic delivery metrics, introduces a new reliability metric, and highlights the impact of team culture, SRE practices, cloud adoption, secure software supply chains, and high‑quality documentation on software delivery and organizational performance.
This article explains why modern agile and DevOps environments need chaos engineering, describes the design and goals of a fault‑injection platform, outlines tool selection, details a five‑step exercise workflow, and shares a real‑world case study that demonstrates improved stability and SRE capabilities.
Over two years, we built a monitoring system covering 200+ services and 700+ instances, evolving from ad‑hoc Nginx logs to a Prometheus‑based observability platform with unified dashboards, automated alerts, and lessons on metric selection, alert fatigue, and root‑cause analysis.
This article introduces a curated list of popular open‑source projects for SRE and DevOps, covering monitoring, deployment, chaos engineering, and reliability tools such as Cloudprober, Istio, Checkov, Litmus, Locust, Prometheus, and more, highlighting their key features and practical use cases.
The article traces the evolution from a rudimentary deployment workflow in a small startup to a mature, Google‑inspired Site Reliability Engineering (SRE) approach, explaining SRE definitions, team duties, error‑budget concepts, key reliability metrics (SLI/SLO/SLA), monitoring implementation with OpenTSDB, and best‑practice alerting rules.
This presentation by NetEase senior SRE Jin Chuan details the current state of NetEase's big data platform, introduces the internally built EasyOps management system, explains a generic Ansible‑based operation framework, describes Prometheus/Grafana monitoring and alerting, and shares practical lessons on network, storage, and cloud migration for large‑scale Hadoop services.
This article shares a practical six‑step framework—from understanding nervousness and defining purpose to designing simple slides, crafting a script, deliberate rehearsal, and handling Q&A—to help technical newcomers deliver confident, audience‑focused presentations at conferences.
This article explores the complexity of IT operations, outlining common misconceptions, essential capabilities, organizational and individual pain points, and presents self‑help strategies such as SRE, DevOps, automation, and AIOps to achieve sustainable, scalable, and intelligent operations within enterprises.
This article summarizes Alibaba Cloud Elastic Compute Service's four‑year SRE journey, covering why ECS created its own SRE organization, the five‑layer SRE framework, standards, automation platforms, empowerment practices, and team‑building insights that can guide large development teams toward reliable, high‑availability operations.
The presentation details NetEase's big data platform operations, covering current usage, the internally built EasyOps control system, a generic service‑operation framework based on Ansible, Prometheus‑Grafana monitoring, configuration management, network and storage optimizations, and lessons learned from cloud migration.
This article outlines the challenges of operating petabyte‑scale big‑data clusters and presents a comprehensive monitoring framework—including basic and upgraded monitoring layers, metric collection, alerting pipelines, and strategies for alarm convergence and grading—to ensure reliable, proactive SRE operations.
The article explains service governance and SRE practices, detailing goals, components, overload handling, capacity planning, and strategies to maintain continuous 24‑hour service reliability while reducing manual toil.
This guide explains how to design and practice a structured incident‑response process—defining problems, applying quick‑recovery steps, analyzing root causes, standardizing solutions, and assigning critical roles—to dramatically reduce production outage duration.
This guide outlines a structured approach to incident response, detailing problem definition, temporary fixes, root‑cause analysis, solution design, implementation, and standardization, while highlighting four critical roles—commander, communicator, rapid‑recovery lead, and diagnosis lead—to ensure swift, coordinated recovery of production services.
This article introduces ten popular open‑source projects for monitoring, deployment, and reliability engineering, detailing each tool's purpose, key features, and how they help Site Reliability Engineers build scalable, highly reliable cloud‑native systems.
This article surveys the most popular open‑source projects for Site Reliability Engineering and DevOps, covering monitoring, deployment, chaos testing, and observability tools such as Cloudprober, Istio, Prometheus, Litmus, and more, highlighting their key features and how they help build scalable, high‑reliability cloud‑native systems.
This article details Xiaomi's implementation of chaos engineering, describing the principles, platform construction using ChaosBlade, a comprehensive fault‑injection workflow, case study results, operational insights, and future plans to enhance system reliability and observability.
This article presents Dada Group's comprehensive approach to elastic capacity management and automated scaling, detailing the challenges faced during traffic spikes, the design of a cloud‑native auto‑scaler, multi‑metric observability, decision‑making logic, execution mechanisms, extreme scaling practices, and future optimization directions.
This article outlines the seven most important Site Reliability Engineering interview questions, explains why they matter, and provides an overview of the upcoming SRE Foundation course that equips professionals with the principles, practices, and tools needed for reliable, scalable systems.
The article provides a comprehensive overview of Site Reliability Engineering (SRE), explaining its origins, definition by Google, required skill sets, typical responsibilities, tools used, and how the role has evolved within DevOps and modern cloud‑native environments.
This article presents a curated list of popular open‑source tools for monitoring, deployment, chaos testing, and reliability engineering, explaining their main features and how they help SREs and DevOps engineers build scalable, highly available cloud‑native systems.
This article distills Bilibili's technical director insights on building high‑service‑quality architectures, covering systematic load‑balancing strategies, sophisticated rate‑limiting mechanisms, robust retry policies, precise timeout controls, and comprehensive approaches to prevent cascading failures in large‑scale systems.
Site Reliability Engineering (SRE) combines software engineering with operations to ensure scalable, highly reliable systems, outlining the collaboration between product development and SRE roles, the software lifecycle, stability value, and practical frameworks for observability, controllability, and best‑practice implementation.
This article presents a comprehensive, step‑by‑step framework for guaranteeing system reliability during high‑traffic promotional periods, covering SRE hierarchy, stability criteria, profiling, monitoring, capacity planning, incident response, and post‑event analysis to help teams build resilient services.
The article explains how Google measures and maintains software quality before release by dividing responsibilities between product teams and SRE, using monorepo, trunk‑based development, daily release candidates, automated testing, performance monitoring, and a Project Health (pH) metric system that tracks productivity, release velocity, reliability, and quality.
This workshop guides participants from installing a single‑node Elastic Stack to deploying a cloud‑native observability platform for a multi‑tier pet‑store application, covering health checks, metrics, logs, APM tracing, SLO/SLI setup, and custom dashboards across local, AWS, and Tencent Cloud environments.
This article explains the role of Site Reliability Engineering (SRE) in bridging product and foundational technology development, outlines the software lifecycle, describes how SRE ensures system stability through controllability, observability, and protection, and provides practical best‑practice checklists and maturity levels for evaluating and improving reliability.
This guide shows how to configure iTerm2 on macOS with custom Ctrl‑arrow shortcuts, set up a minimalist Bash prompt that displays Kubernetes context via kubectx, and manage secure dotfiles using Makefiles, Ansible Vault, and a Docker‑based decryption container for reproducible environments.
This article explores the role of Site Reliability Engineering (SRE) as a bridge between product development and operations, detailing its responsibilities, core principles, lifecycle perspective, stability value, and practical frameworks for controllability, observability, and best‑practice implementation to enhance system reliability.
This article explains the definitions, calculations, and practical importance of MTTR, MTBF, and MTTF for reliability engineering, showing how accurate data and proper metric use enable SRE teams to improve system availability, plan maintenance, and reduce downtime.
The SRE Foundation course introduces site reliability engineering principles, practices, and tools, explaining why perfect reliability is impractical, outlining SRE responsibilities, detailing the curriculum across eight modules, and identifying the diverse professionals—from engineers to managers—who can benefit from mastering reliability, scalability, and automation.
This article presents a comprehensive overview of Site Reliability Engineering (SRE) as shared by Google SRE expert Ramón Medrano Llamas, covering SRE fundamentals, a typical day’s workflow, design principles for massive scale, fault‑tolerant architecture, monitoring, SLI/SLO metrics, redundancy strategies, disaster recovery, and operational best practices.
This article recounts a two‑year journey of building a company‑wide monitoring system, detailing early challenges with openfalcon, the shift to a Prometheus‑Grafana stack, architectural decisions across service, business, and product dimensions, and practical solutions to alert fatigue, threshold setting, and fault isolation.
The article explains how Home Depot (THD) adopted the VALET model—a five‑dimensional SLO language covering Volume, Availability, Latency, Error, and Ticket—to unify communication, automate data collection, and improve reliability across its massive retail and e‑commerce infrastructure.
This article outlines a comprehensive SRE fault‑management framework, covering core responsibilities, stability metrics such as MTBF and MTTR, detailed pre‑, during‑, and post‑incident processes, monitoring, capacity planning, disaster‑recovery, error budgeting, organizational support, and future trends like AIOps and chaos engineering.
This article clarifies common misconceptions about Site Reliability Engineering, outlines the role’s responsibilities, presents the SRE Foundation course syllabus and target audience, and highlights the GOPS 2020 Global Operations Conference where the training is offered.
This article details ByteDance's journey in front‑end engineering, describing the evolution from manual deployments to a fully automated CI/CD pipeline, the creation of a dedicated front‑end deployment platform, and the ongoing development of a comprehensive front‑end R&D platform that integrates DevOps and SRE principles.
This article shares practical SRE insights from years of experience at Alibaba, covering the right mindset, team responsibilities, systematic monitoring, alert management, fault‑handling processes, and resource control to build resilient, high‑availability systems.
This article outlines a comprehensive DevOps operations framework, tracing its evolution from traditional ops to modern automation, detailing business standards, work policies, system integration, and best‑practice norms to achieve high SLA, low cost, and a one‑stop operational platform.
A Huawei maintenance engineer shares a decade‑long journey of turning 24/7 on‑call pain into systematic knowledge management, building comprehensive fault‑handling documentation, automating tools, and guiding the team’s evolution toward SRE practices that dramatically reduce manual effort and improve reliability.
Discover a curated list of ten essential books covering Site Reliability Engineering, performance tuning, AI‑ops, security, DevOps practices, Jenkins pipelines, and the evolution of modern operations, each offering practical insights and real‑world examples to elevate your technical expertise.
This article explains how latency is used as a key indicator for application risk identification, defines slow interfaces, describes why percentile‑based thresholds are preferred over averages, and outlines the architecture, task workflow, and practical optimization strategies for a full‑chain monitoring system in a microservice environment.
This article explains how enterprises adopting SRE can design a comprehensive observability platform—covering metrics, logs, and tracing—while also detailing effective incident response, post‑mortem practices, testing, capacity planning, automation tool development, and user‑experience focus to improve overall operational reliability.
The SRE Foundation course presented at the GOPS 2020 Global Operations Conference in Shenzhen introduces core Site Reliability Engineering principles, practical tools, and certification preparation through eight detailed modules, targeting a wide range of IT professionals and business stakeholders.
This article details Zhejiang Mobile's adaptation of Google‑originated Site Reliability Engineering to a telecom environment, outlining a three‑layer capability framework, standardized processes, integrated platforms, and measurable outcomes that demonstrate how agile SRE practices can boost reliability and scalability in traditional industries.
This article explains the evolution from traditional C/S to B/S architectures, compares DevOps and SRE principles, discusses their roles in the container and cloud eras, and showcases StarRing's TDC platform that integrates automated pipelines, monitoring, and deployment for efficient software delivery.
In this detailed Q&A, a Meitu SRE leader explains the relationship between DevOps and SRE, shares practical advice on team composition, monitoring, alerting, fault‑prevention design, and provides step‑by‑step guidance using Grafana, draw.io, and other tools to help organizations build reliable services.
The article chronicles Zheng Yangfei’s rise from a double‑eleven intern to leader of Ant Group’s cloud‑native capacity team, detailing the evolution of large‑scale load‑testing, the challenges of autoscaling in financial‑grade systems, and the team’s shift toward platform‑driven, risk‑aware engineering.
During the pandemic’s “停课不停学” surge, Tencent Classroom tackled a 120‑fold traffic jump by rapidly deploying Grafana dashboards, Kibana logs, internal Moniter and cloud monitoring tools, establishing a three‑layer feedback‑alert‑on‑call model, and now plans automation, unified visualizations, and chaos‑engineering to further boost observability and service reliability.
Drawing on Google SRE principles, Bilibili’s technical director outlines a systematic, cloud‑native framework for high‑quality service architecture during traffic peaks, covering frontend and internal load balancing, distributed rate limiting, controlled retries, fail‑fast timeouts, and comprehensive failure‑mitigation strategies.
This article explains why Site Reliability Engineering (SRE) emerged, outlines its core responsibilities, required skill set, and how it addresses reliability challenges through decoupling, SLO‑driven monitoring, and scenario‑based drills, while highlighting key observations and focus areas for modern operations teams.
This article explains how Linux kernels receive network packets—from NIC hardware interrupts through NAPI polling, kernel TCP/IP processing, and finally to user‑space sockets—while also covering interrupt handling, buffer tuning, and performance‑optimizing techniques for SREs.
Zhejiang Mobile’s IT Operations team announced a strategic shift from reactive ticket‑driven maintenance to a proactive, AI‑powered AIOpsDev model, establishing new departments, adopting SRE practices, and leveraging cloud‑native technologies to dramatically improve efficiency, reliability, and digital transformation.
The article describes how Dada rebuilt its log processing pipeline using Kubernetes mixed deployment, Filebeat for automated collection, Storm for efficient parsing, and Elasticsearch cold/hot nodes to handle over 130 billion daily log entries and 300TB storage.
This article introduces Flyway as a database migration tool, explains its working principle, directory and naming conventions, supported databases, and provides detailed step‑by‑step instructions, best‑practice guidelines, and troubleshooting tips for safely managing MySQL schema changes in production environments.
This article examines the inherent tension between operations and development, explains Google’s error‑budget and SLO approach, and shares practical DevOps, on‑call, automation, and talent strategies that help ops teams improve efficiency while maintaining product reliability.
This article examines the tension between operations and development teams, explains Google's SRE error‑budget model, and shares practical reflections on engineering ops, on‑call rotation, automation, and talent development to achieve a sustainable balance between product stability and rapid innovation.
This article explains the origins of Site Reliability Engineering (SRE), outlines the responsibilities of SRE teams, and details Alibaba Cloud’s ECS SRE practices—including capacity planning, performance optimization, full‑stack stability governance, automated release pipelines, on‑call processes, and the core principles and mindset that guide modern SRE work.
The talk outlines the unique operational challenges of a fast‑growing e‑commerce platform—including massive scale, frequent changes, cost pressures, and the trade‑off between speed and stability—and describes how the SRE team uses automation, capacity planning, and process engineering to deliver reliable, efficient services.
This article explains how Alibaba's Elastic Compute Service (ECS) SRE team tackled massive traffic, database bottlenecks, alert overload, and resource inconsistencies by establishing a full‑stack reliability organization, upgrading core components, automating pipelines, and instituting rigorous monitoring, incident response, and change‑management processes.
This article presents Google SRE expert Ramón Medrano Llamas’s comprehensive guide on designing, operating, and maintaining large‑scale, highly available software systems, covering SRE fundamentals, daily workflows, scalability strategies, fault‑tolerant architecture, monitoring, and operational best practices.
Xiami’s SRE team overhauled its monitoring system by categorizing alerts, introducing fault, generic, and basic monitoring, optimizing alert paths with stream processing, and leveraging Alibaba’s traffic scheduling platform, dramatically reducing daily noise from thousands of alerts to a manageable few hundred critical notifications.
This article examines how Google SRE limits weekly alerts to ten, compares it with typical Chinese internet operations teams, and provides practical strategies—including on‑call scheduling, alert escalation, automation, dashboard optimization, and team management—to dramatically reduce alert volume and improve incident response.
This article explores how to construct a scalable operations framework by integrating concepts such as SRE, DevOps, AIOps, and continuous improvement, addressing organizational challenges, process standardization, tool automation, and the shift from reactive firefighting to proactive, value‑driven management.
Network engineers often go unnoticed until outages, so this guide explains how to make network status transparent through device availability checks, log and traffic monitoring, SNMP error tracking, and automation scripts—leveraging Python, syslog servers, and northbound APIs—to reduce troubleshooting time and prevent incidents.
The article explains why DevOps and SRE engineers top the 2019 StackOverflow IT job popularity list, outlines their responsibilities, career prospects, required skills, and provides practical advice for aspiring professionals.
This article examines Google’s corporate values, development history, culture, and detailed DevOps and Site Reliability Engineering practices—including continuous delivery, SRE responsibilities, and Google Cloud Platform CI/CD tools—to illustrate how the company achieves 24/7 reliable service deployment at massive scale.
The article details how Alipay’s technical risk team, led by researcher Chen Liang, evolved from early scalability work to a full‑stack SRE organization, created the TRaaS risk‑defense platform and integrated AIOps to achieve near‑five‑nine availability and automated self‑healing for its financial services.
In this talk, senior NetEase operations engineer Gu Xianjie shares a decade‑long journey tackling technical debt, rapid product growth, and on‑call pain points, describing the evolution from manual scripts to automated platforms, service‑oriented tools, DevOps/SRE practices, and cloud‑native strategies that boosted efficiency and reliability.
This article explains how network engineers can use lightweight monitoring solutions, log analysis, traffic and error tracking, and custom automation scripts to gain visibility, reduce troubleshooting time, and safely automate routine network tasks in enterprise environments.
This article explains how to approach monitoring for a newly introduced system by focusing on white‑box metric collection, distinguishing basic and business metrics, outlining common collection methods, and detailing Google SRE's four golden indicators—error, latency, traffic, and saturation—to guide effective observability.
This article explains the differences between white‑box and black‑box monitoring, outlines common data‑collection techniques for both basic and business metrics, and details Google SRE’s four golden indicators—error, latency, traffic, and saturation—to help engineers design effective monitoring solutions.
Didi’s data‑driven Risk Quantification Platform assigns numeric Change Credit and Monitoring Health scores to deployments, alerts and core services, turning operational best‑practice adoption into a competitive game that has raised scores, cut incident rates despite higher change volume, and paves the way for broader risk‑management across the organization.
This article systematically explains how to enhance e‑commerce platform availability by implementing both black‑box monitoring to detect functional failures and white‑box monitoring to pinpoint root causes, detailing core order‑process metrics, common issues, mitigation strategies, and illustrative Grafana dashboards.
Ant Financial’s internal red‑blue technical attack‑defense program, driven by a dedicated blue team and SRE‑based red team, continuously probes system weaknesses, refines fault‑injection tools like Awatch, and evolves high‑availability and self‑healing mechanisms to strengthen risk control and operational reliability.
The article analyzes how Google responded to a testing crisis by empowering a visionary leader, establishing supportive structures, encouraging innovation, and persisting over years to embed a quality‑centric culture that eventually led to decentralized testing, SRE adoption, and a shift toward test‑design engineers.
This article provides a comprehensive, operations‑focused overview of Elasticsearch monitoring, covering tool selection, key metrics for black‑box and white‑box monitoring, common issues discovered through alerts, and practical optimization recommendations to ensure high availability of ES clusters.
Google’s production environment spans a global edge network, massive data centers, sophisticated job scheduling with Borg, distributed storage systems like Bigtable and Spanner, and comprehensive monitoring, illustrating how user requests traverse multiple layers—from ISP to edge, GFE, load balancers, and finally to services.
The article explains Google’s end‑to‑end production infrastructure—including the edge network, data‑center hierarchy, Borg‑based cluster management, storage systems like Colossus and Spanner, monitoring with Borgmon, inter‑task RPC via Stubby, and the code‑to‑production pipeline using Piper, Blaze, Rapid, and Sisyphus—illustrating how requests travel from users to services in milliseconds.