A nine‑hour outage of the popular travel app exposed how reliance on AI‑generated code can mask deeper failures in disaster‑recovery planning, incident response, and engineering rigor, reminding developers that high availability depends on disciplined practices rather than tools.
Airbnb’s Sitar platform demonstrates how a modern dynamic configuration system can provide safe, reliable, and flexible runtime changes through a Git‑centric workflow, multi‑tenant control and data planes, staged rollouts, rapid rollback, and local caching, balancing developer agility with operational stability.
In a landscape crowded with OLTP, OLAP, HTAP, NewSQL and cloud‑native options, this article explains why enterprises must look beyond performance and assess a database’s five‑dimensional long‑term service capability to ensure sustainable growth and low migration risk.
This article shares a year‑long, hands‑on experience of improving backend alert governance at Tencent Meeting, covering why alerts are hard, designing segmented error codes, building unified alert policies, driving team silence‑up, measuring progress, and the tools that make the process sustainable.
This article walks through the challenges of traditional traffic replay, explains the design and benefits of a conventional replay system, and then details how integrating large language models can automate data preparation, script generation, and validation to make backend testing more accurate, scalable, and efficient.
The June 2025 Google Cloud outage, caused by an untested code change that triggered a NullPointerException, crippled over 70 core services worldwide, prompting a rapid technical fix, public apology, and industry‑wide reflections on cloud stability, fault tolerance, and deployment practices.
Baidu Netdisk engineers protect its billion‑user service from cascading failures by deploying dynamic circuit‑breaker overload control, priority‑based traffic isolation, request‑validity filtering, socket‑level disconnect detection, and unified timestamp handling, a combination that dramatically reduces avalanche incidents and boosts overall availability.
This article analyzes Baidu Netdisk's anti‑avalanche architecture, explaining how avalanche cascades occur in high‑concurrency services and detailing practical prevention, blocking, and mitigation techniques such as dynamic circuit breaking, traffic isolation, request‑validity checks, and socket‑level detection to maintain system reliability.
This article analyzes a severe service‑availability drop caused by Log4j2 asynchronous logging bottlenecks, explains how configuring log4j2.asyncQueueFullPolicy=Discard and log4j2.discardThreshold=ERROR mitigates the issue, details the investigation steps, performance tests, and provides practical recommendations for robust backend logging.
This article explains how Netflix extended its priority load‑shedding strategy from the API gateway to individual services, detailing the classification of user‑initiated versus pre‑fetch requests, the implementation of partitioned concurrency limiters, CPU‑ and I/O‑based shedding, test results, and real‑world impact on availability.
This article explains the definitions and relationships of SLA, SLO, and SLI, shows how to set realistic targets, presents service‑level grading, alert‑noise reduction techniques, and practical examples to help teams prepare for large‑scale events such as the 11.11 promotion.
This article explains the core concepts of SLA, SLO, and SLI, demonstrates how to set realistic service level objectives, manage alert noise, and apply practical examples—including API, MQ, and scheduled task monitoring—to improve system reliability and performance during high‑traffic events like 11.11 promotions.
This article explains why service stability is critical, outlines the importance and key factors of synthetic monitoring, provides practical guidelines for implementing it, and then describes fault‑drill concepts, benefits, processes, and common cloud‑native tools to proactively discover and mitigate failures in micro‑service environments.
This article describes a real‑world incident where rapid pod scaling caused order‑submission failures in a serverless e‑commerce platform, analyzes the root causes, and presents a Go‑based graceful‑shutdown solution—including ASyncClose, SyncClose, and ForceSyncClose modes—implemented in the open‑source GS library to help developers reliably terminate services.
The article examines the rapid growth of China's cloud market, the frequent "cloud collapse" incidents, their root causes in governance failures, and presents practical cloud governance measures along with an overview of the new industry standard for enterprise cloud governance capability maturity.
The article analyzes a recent Tencent Cloud outage, detailing the specific API failures, contrasting the limited impact on IaaS services with widespread PaaS disruptions, and argues for multi‑cloud redundancy while critiquing sensationalist news and outdated status‑page expectations.
On April 8, a Tencent Cloud API outage caused console login failures for nearly 2,000 customers, affecting several dependent services for 87 minutes, and the detailed root‑cause analysis and subsequent improvement actions are presented to enhance system resilience and change management.
The article outlines how game QA and third‑party providers can improve cooperation by aligning basic performance concepts such as TPS, QPS and concurrency, selecting appropriate rate‑limiting strategies, establishing precise monitoring and alerting, and preparing clear incident‑response and delivery standards.
By separating test‑traffic triggering from result verification, Didi’s ride‑hailing backend uses live‑traffic inspection and replayed offline tests with bucketed validation rules to achieve near‑zero‑cost, full‑coverage QA, catching hundreds of bugs annually and dramatically improving service reliability for drivers and passengers.
This document defines the online incident severity grading system, outlining fault categories, influencing factors such as business metrics, capital loss, user impact, and public opinion, and presents detailed P0‑P3 grading rules with tables for capital‑based, C‑end, and B‑end user classifications.
Through a five‑year backend journey—from building a solo startup site and mastering Java, to handling high‑traffic services at Sina Weibo, and now developing B2B merchant tools at Alibaba Taobao—the author shares lessons on scalable architecture, automated deployment, aligning tech with business, proactive problem‑solving, code quality, teamwork, and career health.
This article explains why high availability is crucial for backend services and outlines ten practical design principles—including dependency control, avoiding single points, load balancing, isolation, rate limiting, circuit breaking, async processing, degradation, gray release, and chaos engineering—to help developers create resilient APIs.
The article describes a root‑cause localization algorithm implemented in vivo’s monitoring platform that automatically analyzes latency spikes by splitting service timelines, computing variance, clustering results with K‑means, and recursively tracing downstream services, achieving over 85 % accuracy for dependency failures while still requiring human verification and outlining future AI‑driven enhancements.
This article details the operational mindset, stability framework, and transformation journey of the Zuoyebang infrastructure team, covering service lifecycle management, standardization, cloud‑native architecture, multi‑active deployment, incident pre‑plan platforms, traffic scheduling, monitoring, capacity planning, and future directions for SRE service‑orientation.
This article breaks down WeChat’s 2018 overload control system for massive microservices, explaining the problem of service overload, detection via average waiting time, and a multi‑level priority‑based mitigation strategy that dynamically adjusts admission thresholds to keep billions of daily requests stable.
This article details Bilibili's end‑to‑end full‑link load‑testing solution for its live‑stream gifting service, covering industry alternatives, the chosen architecture, a three‑stage automated testing framework, link analysis, configuration, validation, and practical case studies to ensure system stability under massive traffic spikes.
This article explains the fundamentals of Site Reliability Engineering, outlines a complete SRE workflow from prevention to post‑mortem, details key availability metrics and golden indicators, examines how technical debt arises and can be mitigated, and describes the tooling and practices needed to keep large‑scale services healthy.
This talk recounts how the Good Doctor platform tackled severe online incidents by launching the DOA project, then a service risk governance initiative that identifies, quantifies, and mitigates latency‑related risks through metrics‑driven development, dependency analysis, middleware reliability, and a dedicated risk‑management platform.
This guide outlines essential incident management principles—prioritizing business recovery and timely escalation—and presents practical methodologies such as restart, isolation, and downgrade, while detailing user impact handling, organizational roles, and post‑mortem best practices.
This article outlines Youku's end‑to‑end service‑side quality assurance framework, detailing the factors that affect quality across the development lifecycle, the automated testing practices integrated into the release pipeline, the platform capabilities built for data collection and replay, and the metrics used to measure improvements in reliability and development efficiency.
This article analyzes the concept of stability governance, outlines its five fault‑management sub‑domains, examines the pain points of an electronic waybill service, and presents a comprehensive three‑phase strategy—prevention, perception, reach, mitigation, and post‑mortem—backed by concrete implementation steps in availability, monitoring, and online emergency handling.
This guide outlines essential incident‑response principles—prioritizing business restoration and timely escalation—while detailing practical methods such as restart, isolation, and degradation, and explains how to organize response teams and conduct thorough post‑incident reviews.
This guide outlines essential incident management principles—prioritizing business restoration and timely escalation—followed by detailed methodologies such as restart, isolation, and degradation, and explains role responsibilities, user impact handling, and post‑incident summarization for continuous improvement.
The July 13, 2020 GitHub outage, triggered by load‑balancer misconfiguration, a database connection error during partitioning, and a network‑config mistake, sparked worldwide developer panic, highlighted reliability concerns, and revealed challenges in scaling cloud infrastructure amid the pandemic.
Drawing on five years of incident reviews, the article defines a comprehensive stability framework for backend services—mandating timeout hierarchies, weak dependencies, service-discovery integration, staged gray releases, robust monitoring, capacity planning, and strict change management—while cataloguing common anti-patterns such as over-aggressive circuit breaking, static retries, improper timeouts, tight coupling, and insufficient isolation, and urging regular rehearsal of these practices.
The article explains why distributed systems must stay alive despite upstream or downstream failures, emphasizing rate limiting and circuit breaking as essential practices to prevent fault propagation and ensure service reliability, and it invites developers to assess their own safeguards.
This article outlines a systematic approach to dependency governance in high‑traffic backend services, covering service classification, rate limiting, Dubbo, HTTP, database, and message‑queue management to enhance availability, reduce failure impact, and improve overall system stability.
The article presents a systematic capacity‑estimation methodology that links service traffic to order volume, uses CPU‑Idle as a primary metric, predicts traffic growth and upper‑bound limits, validates predictions with load‑testing, and provides scaling recommendations while noting limitations of the CPU‑Idle baseline.
The article describes JD's traffic replication and replay platform, explaining its background, the concepts of traffic copying and replay, detailed platform architecture and features, normalised load testing workflow, dynamic regression testing, operational results, current limitations, and future improvement directions.
The article explains why enterprise application monitoring systems are essential, outlines their core components such as Trace, Log, Metric, and Report, discusses distributed tracing techniques, and describes how these insights are designed to aid developers, testers, and operations engineers in performance tuning and fault diagnosis.
This article explains the essential requirements for achieving 99.99% service availability—consistency, eliminating single points, placement groups, traffic isolation, same‑city active‑active, N+1 redundancy, and multi‑region active‑active—illustrated with a step‑by‑step Yum repository service case study and evolving architecture diagrams.
The article shares a comprehensive, experience‑driven guide on building fault‑tolerant systems—covering retry mechanisms, dynamic node removal, timeout settings, service degradation, decoupling, and business‑level safeguards—to enable a platform that scales from millions to billions of daily requests without relying on manual fire‑fighting.
This article shares practical strategies for avoiding system failures by doubting third‑party services, protecting against misuse by consumers, and strengthening internal design through solid API practices, resource limits, and disciplined coding principles.
This article outlines Youzan's end‑to‑end online incident management process—from fault detection and coordination through root‑cause analysis, recovery, review, and actionable JIRA tracking—highlighting practical workflows, data analysis, and continuous improvement practices for reliable service delivery.
This article details Meituan's practical approaches to boosting account service reliability, covering MTBF/MTTR metrics, business‑level monitoring, flexible availability with circuit‑breaker patterns, cross‑region active‑active deployment, data synchronization techniques, and the measurable performance gains achieved.
This talk outlines the five major challenges faced by Tencent's SNG component operations—geographic distribution, HTTPS certificate management, massive device failures, long‑term maintenance, and large‑scale scaling—and describes the underlying architecture, operational principles, and practical techniques used to automate and reliably support millions of users during peak events.
This article examines the pitfalls of naive retry mechanisms, explores active‑standby service switching, dynamic removal of unhealthy nodes, proper timeout configuration, and anti‑reentrancy strategies to improve system reliability and prevent cascading failures in large‑scale backend operations.
The SNG Operations team shares the five critical challenges of managing tens of thousands of domains, certificates, server failures, automation, and rapid scaling during peak events, and outlines the practical strategies they used to ensure reliable, near‑real‑time service delivery.
This article details Ele.me's rapid expansion challenges and shares a three‑stage technical operations journey—fine‑grained division, stability maintenance, and efficiency gains—highlighting real incidents, monitoring upgrades, capacity testing, and practical insights for reliable large‑scale delivery platforms.
The article outlines Meituan‑Dianping’s approach to front‑end service availability for its checkout system, defining availability across code, static resources, and network links, measuring failure duration, identifying typical bugs, and implementing a three‑stage assurance strategy using people processes, engineering tools, lightweight technology choices, and concrete practices such as TypeScript adoption, automated testing, health‑checks, DNS protection, and post‑incident monitoring.
The article explains how LinkedIn’s Redliner system automatically measures service capacity and performs low‑impact, production‑traffic stress tests to identify bottlenecks, guide resource allocation, and support proactive capacity planning and performance regression testing.
The article shares Tencent’s practical fault‑tolerance journey for a billion‑scale activity platform, covering retry strategies, automatic removal of faulty nodes, timeout tuning, business‑level safeguards, service degradation, and decoupling techniques that together reduce manual firefighting and improve system resilience.
This article explains the definitions of service, SLI, SLO, and SLA, outlines how to choose and measure appropriate indicators, shares best practices for setting and improving SLOs, and shows how SLAs combine objectives with consequences to manage service reliability.
This article explores how Google Site Reliability Engineers manage service reliability by balancing risk, cost, and business goals, using metrics like unplanned downtime, availability formulas, and risk tolerance to set realistic SLOs for both consumer and infrastructure services.
Uber’s adoption of microservices showcases both the flexibility of using multiple languages and independent release cycles, while also exposing challenges such as duplicated effort across teams, type‑unsafe JSON interfaces, and the need for rigorous failure testing, offering valuable lessons for large‑scale system design.
The article examines how rapid mobile internet growth reshapes software quality expectations, urging QA engineers to balance bug prevention, end‑to‑end service reliability, and user‑centric product excellence across evolving B2C scenarios.
An experienced backend engineer shares practical strategies to prevent service failures, covering third‑party distrust, user‑side safeguards, robust API design, traffic limiting, resource management, and architectural best practices such as single‑responsibility and avoiding single points of failure.
