On December 5th Alipay experienced a brief outage that sent users into panic, prompting a humorous recount of personal losses, meme images, and a reminder of the critical importance of high‑availability architecture and disaster‑recovery planning for large‑scale financial services.
JD.com's Retail Technology and Data Platform orchestrated a full‑chain, four‑entry‑point stress test for the 11.11 shopping festival, introduced an upgraded Forcebot traffic‑recording tool, and implemented a "Quality Month" with ten safeguards to ensure system stability and prevent incidents during the massive sales event.
At QCon Beijing 2019, senior architect Wang Junxiang shared Tong‑Cheng Yilong’s end‑to‑end degradation‑plan architecture, covering system design, data collection, metric computation, resource recovery, link‑level pre‑plan management, fault diagnosis, strategy extensibility, and high‑availability platform construction, offering practical insights for complex distributed systems.
The article explains how anticipating and engineering for diverse failure scenarios—from hardware faults and software bugs to traffic spikes and external attacks—can dramatically improve system reliability, reduce downtime, and protect business continuity in modern distributed and cloud environments.
This talk shares a senior director’s experience building a hybrid multi‑cloud infrastructure for a game company, covering stability, efficiency, cost challenges, design‑for‑failure principles, standardization, resource automation, and the cultural and organizational factors that affect successful cloud operations.
This article outlines Qunar's operational strategy for reducing failures and extending uptime through precise fault detection, rapid recovery, and AI-powered predictive health management, detailing the evolution of their OPS processes, practical implementations, and future challenges in applying PHM to internet services.
The article explores how unpredictable black‑swan events, the butterfly effect, Murphy's law, rapid business growth, technical debt, tool choices, complex domains, documentation, and leadership all combine to threaten software stability, and proposes agile, systematic, and quality‑centric approaches to mitigate these risks.
The article details Didi's end‑to‑end load‑testing strategy—including online environment testing, data isolation with virtual orders, trace‑based traffic marking, and a distributed virtual driver/passenger tool—describing its design, deployment stages, findings, and future reliability applications.
The article recounts the Forbidden City’s first night‑time Lantern Festival event, the overwhelming demand that caused the museum’s ticketing website to crash, and includes an interview with a senior operations engineer who explains the causes of such overloads and outlines rapid mitigation and scaling strategies.
From its humble 2009 launch to the 2018 tenth anniversary, Alibaba’s Double 11 shopping festival evolved through relentless technical challenges—system crashes, CDN bottlenecks, over‑selling bugs, and massive load‑testing innovations—offering a decade‑long case study in operations, scalability, and resilience for large‑scale e‑commerce platforms.
From the humble 2009 launch of Double 11 to the massive, cloud-native, multi-region architecture of 2018, Alibaba’s engineers chronicle yearly technical hurdles—traffic spikes, system crashes, CDN limits, over-selling, and the evolution of stress-testing, capacity planning, and operational safeguards that turned the shopping festival into a global engineering showcase.
This article explains why distributed caching is essential for high‑concurrency systems, outlines typical use cases, compares Memcached and Redis across features and performance, and discusses common problems such as cache avalanche, penetration, warm‑up, update strategies, and degradation.
This article shares practical methodologies for designing, deploying, and maintaining systems that can reliably operate for ten years, covering goal setting, holistic design considerations, carrier and data‑center choices, active‑active architecture, server and platform selection, monitoring, and continuous personal improvement.
This article shares practical methodologies for building software systems that remain stable for a decade, covering goal setting, holistic design, operator and data‑center choices, cross‑region active‑active challenges, server and platform selection, comprehensive monitoring, and the importance of continuous personal improvement.
This article details how JD's Life Technology Service Platform prepared for the 6.18 shopping festival through system upgrades, stress testing, monitoring enhancements, and emergency drills to ensure high availability, performance, and reliable service for users.
This article analyzes the rapid growth challenges of Meituan‑Dianping's core payment flow, explains key availability metrics such as MTBF and MTTR, and presents a comprehensive set of architectural, operational, and tooling strategies—including dependency decoupling, timeout tuning, circuit breaking, and full‑link stress testing—to achieve stable, fault‑tolerant transactions.
This article presents a comprehensive case study of Baidu Nuomi's online load‑testing methodology for high‑traffic marketing events, covering capacity estimation, test planning, execution, anti‑attack measures, platform architecture, and lessons learned to ensure system reliability and performance under peak loads.
This article explains how setting low timeouts for non‑core services, decoupling and physically isolating micro‑services, separating light and heavy workloads, and implementing automated configuration checks together enhance system reliability and reduce both technical and human errors in high‑traffic environments.
This article presents a comprehensive list of 85 practical operations rules covering capacity planning, monitoring, automation, security, documentation, budgeting, team management, and incident handling, offering actionable guidance for building reliable, scalable, and efficient IT infrastructure.
This article shares practical SRE‑based principles for diagnosing and resolving online incidents, emphasizing systematic investigation, gathering clues, and prioritizing service restoration over immediate root‑cause identification to make troubleshooting less mystical and more effective.
This article explains how load balancing and reverse proxy mechanisms such as Nginx, Consul, and Hystrix work together with various isolation strategies—including thread, process, cluster, data‑center, and resource isolation—to improve system reliability and scalability in large‑scale web architectures.
The article explores how adopting a "don't trust" mindset—through rigorous input validation, defensive coding, thorough testing, gradual rollouts, and comprehensive monitoring—helps build resilient, high‑performance systems and avoid common pitfalls in development and operations.
This article explains common load‑balancing strategies—round‑robin, random, minimum response time, minimum concurrency, and hash—detailing their principles, advantages, drawbacks, and mathematical reliability analysis, including probability formulas and visual illustrations to help choose the most fault‑tolerant approach for distributed systems.
This article outlines a comprehensive PDCA‑based methodology for e‑commerce platforms to proactively prevent issues, quickly detect anomalies, and execute rapid decisions during large‑scale promotions, covering system goal definition, performance evaluation, capacity planning, SLA management, and team/process maturity.
The QQ Game Spring Festival Red Packet system was engineered with multi‑data‑center deployment, global load balancing, layered overload protection, flexible critical‑path redundancy, three‑dimensional monitoring, and extensive rehearsal testing, delivering high‑availability and fault‑tolerant service even under extreme traffic spikes.
This guide teaches operations engineers a four‑step “Wang‑Wen‑Wen‑Qie” methodology—observing system health, listening via alerts, questioning changes, and pulse‑checking with profiling tools—to rapidly diagnose and resolve high‑impact incidents while maintaining clear communication and post‑mortem learning.
This article explains how to design and operate feature toggles in Java applications, covering single‑instance implementation, cross‑instance synchronization via a meta‑server or Diamond, handling composite switches, avoiding security pitfalls, and automating degradation and upgrade based on runtime metrics.
This article explores comprehensive service degradation techniques—including automatic and manual switchovers, read/write and multi‑level fallback strategies, and practical examples like timeout, failure count, and traffic throttling—to ensure core functionality remains available during traffic spikes or component failures in high‑concurrency systems.
This article provides a comprehensive guide to the Linux Test Project (LTP), covering its purpose, supported architectures, directory layout, installation steps, test categories, execution scripts, stress‑testing commands, result analysis, and troubleshooting tips for improving kernel stability and reliability.
This article explores practical service‑degradation techniques—including automatic and manual switches, read/write fallback, and multi‑level strategies—to ensure core functionality remains available during traffic spikes, failures, or resource constraints in high‑concurrency systems for.
Twitter keeps its platform stable during massive traffic spikes by regularly performing large‑scale stress and extreme tests, analyzing performance metrics, and maintaining detailed contingency plans that guide rapid response to unexpected events such as the record‑breaking “Sky City” incident.
The article explains how proper cache usage can dramatically improve resource utilization, response time, and reliability in high‑concurrency front‑end and back‑end systems, while also addressing cache hit‑rate optimization, data consistency, and mitigation techniques for cache penetration and avalanche scenarios.
This article shares the author’s experience building fault‑tolerance for Tencent’s activity operations platform, covering retry strategies, automatic removal of unhealthy machines, timeout tuning, asynchronous processing, anti‑replay mechanisms, service degradation, service decoupling, and business‑level safeguards to reduce manual alarm handling and improve system robustness.
The article examines system reliability concepts such as availability, MTBF, MTTR, and outage classifications, analyzes the Alipay service interruption, discusses various redundancy and failover strategies, and explores YunOS reliability testing and design practices to improve overall system robustness.
The article explains why service degradation is essential for large‑scale shopping events, outlines its different dimensions such as page, business module, and remote service downgrade, and describes both manual and automatic implementation methods to maintain system availability under heavy load.
This article presents a comprehensive set of logging best‑practice rules—from defining log levels and classifications to using RequestIDs, monitoring alerts, and managing log size—aimed at improving system reliability, troubleshooting speed, and operational efficiency.
The article examines the differing challenges faced by development and operations teams, explains key concepts of system performance, scalability, stateless design, and session replication, and offers practical principles to align both sides for reliable, cost‑effective software delivery.
