Tagged articles
incident-analysis
28 articles · Page 1 of 1
This article analyzes three real production incidents caused by broken TraceId propagation, uncontrolled log volume, and sensitive data exposure, then presents a comprehensive microservice logging framework covering log levels, mandatory fields, exception handling, dynamic level governance, cold-hot storage tiering, and a production checklist.
Cost OptimizationLoggingTraceId
0 likes · 20 min read


21CTOAug 20, 2026 · Operations
GitHub’s nearly eight‑hour outage was caused by a saturated load balancer in its US‑central data center, a mis‑configured auto‑scaling policy that ignored Istio sidecar concurrency limits, and an optimistic retry loop in VS Code that amplified traffic tenfold, prompting emergency mitigations and a staggered recovery.
Auto ScalingGitHubIstio
0 likes · 4 min read

This article compiles twelve high‑profile internet service failures from 2025, detailing each incident’s description, micro‑scenario, technical root cause, and risk perspective, and extracts actionable lessons on infrastructure resilience, change management, and security‑aware operations.
Internet OutagesOperationsSRE
0 likes · 20 min read

On the night of November 18 2025, Cloudflare suffered a three‑hour core failure that crippled roughly half of the internet, and this article details the timeline, global impact, root cause in a ClickHouse permission change, and the remediation steps taken to restore service.
Bot ManagementCDNCloudflare
0 likes · 10 min read

On the night of November 18 2025, Cloudflare suffered a worldwide outage that crippled services like ChatGPT, X, Spotify, and major gaming platforms, and a detailed post‑mortem reveals that a ClickHouse permission change caused an oversized bot‑management configuration file to crash edge nodes.
Bot ManagementCDNClickHouse
0 likes · 9 min read

During the Double‑11 shopping festival, a leading e‑commerce platform suffered a P0 outage when its Eureka service‑discovery cluster overloaded, triggering a full‑chain failure that lasted 2 hours 42 minutes and caused losses exceeding 1.2 billion yuan; the article dissects the timeline, root causes, capacity mis‑planning, monitoring gaps, and remediation strategies.
EurekaJavaService Discovery
0 likes · 34 min read

A massive outage on June 12, 2025 crippled Google Cloud, AWS, and Azure, exposing the hidden risks of multi‑cloud architectures as a simple NullPointerException cascaded into a global digital infrastructure failure.
AWSAzureGoogle Cloud
0 likes · 4 min read

The article dissects Alipay’s rare P0 incident on January 16 2025, explaining how a misconfigured marketing template triggered a 20% discount for all transactions, detailing the rapid five‑minute fix, estimating the financial loss at roughly 14 million yuan, and outlining operational lessons and accountability.
Operationsdeployment riskfinancial loss
0 likes · 11 min read

On December 11, 2024 OpenAI’s Kubernetes cluster suffered a four‑hour outage after a newly deployed telemetry service generated massive API traffic from every node, overwhelming the kube‑apiserver, breaking DNS‑based service discovery, and exposing gaps in control‑plane monitoring and break‑glass mechanisms, prompting critical questions about component behavior and configuration.
API overloadDNSKubernetes
0 likes · 8 min read

On Double 11, Alipay suffered a multi‑hour outage that prevented payments, Yebao withdrawals, and Huabei repayments, prompting an official apology that blamed a partial failure of its system message library—a critical database for storing and routing system messages—highlighting hardware, software, network, and data factors behind the incident.
AlipaySystem Message LibraryTechnology Failures
0 likes · 8 min read

This article recounts a developer's journey from a fresh graduate to a senior backend engineer, detailing two real‑world incidents—a pseudo‑memory‑leak in a C++ service and a uint overflow that caused traffic bursts—showing the analysis steps, code fixes, and lessons learned for reliable backend development.
C++Memory Leakincident-analysis
0 likes · 19 min read

Google Cloud mistakenly deleted data and backups for Australian pension fund UniSuper, causing over 600,000 members to lose access for more than a week, and the incident highlights the risks of single‑provider reliance, the importance of robust backup strategies, and the growing relevance of hybrid and multi‑cloud architectures.
Google CloudMulti-cloudbackup
0 likes · 5 min read

The article explains the design and functionality of an AI‑enhanced event intelligent analysis system that automates fault identification, analysis, and remediation in data‑center operations, detailing its architecture, integration with monitoring, CMDB, ITSM, big‑data platforms, and the AI techniques that enable automatic modeling, clustering, and knowledge‑base retrieval.
AIautomationbig data
0 likes · 18 min read

An in‑depth review of Didi’s 12‑hour P0 outage reveals how a mistaken Kubernetes version downgrade during an in‑place upgrade caused master node failure, discusses cluster isolation, upgrade strategies, and the role of cost‑cutting pressures, offering practical lessons for large‑scale operations.
KubernetesOperationscluster upgrade
0 likes · 7 min read

The article reviews Didi’s 12‑hour P0 outage caused by a Kubernetes upgrade failure in a massive cluster, discusses the root causes, and proposes general solutions such as federation, careful upgrade planning, and multi‑master designs to avoid similar incidents.
Cluster ScalingKubernetesincident-analysis
0 likes · 8 min read

The article reviews Didi's late‑November 2023 service disruption, detailing the timeline of failures, official apologies, and expert analyses of six possible technical causes—including software bugs, server issues, third‑party failures, DDoS, other attacks, and ransomware—while highlighting the role of a Kubernetes upgrade and cost‑cutting pressures.
DidiOperationscloud-native
0 likes · 7 min read


LouZaiNov 29, 2023 · Industry Insights
The article reviews Didi’s massive November 27‑28 outage, examines reported symptoms, explores likely technical and organizational reasons such as infrastructure failures and cost‑cutting measures, and reflects on the impact of recent layoffs and server‑cost reductions on service reliability.
Cloud MigrationDidicost reduction
0 likes · 6 min read

The article examines Vipshop's massive March 2023 outage caused by an IDC cooling failure, critiques superficial PPT‑driven reliability claims, and offers practical SRE insights on fault drills, true multi‑active architectures, and how ops teams can gain influence despite budget constraints.
Multi-ActiveOperationsSRE
0 likes · 7 min read

A government information system suffered a week of instability, including service deadlocks, Tomcat memory overflows, and load‑balancing failures, prompting a deep forensic analysis that uncovered database lock‑ups, faulty front‑end loops, inadequate monitoring, and misconfigured logging, leading to concrete remediation steps and lessons for future reliability.
OperationsTomcatincident-analysis
0 likes · 21 min read

A production incident on an elastic‑cloud deployment revealed that setting the JVM Metaspace limit to 64 MiB, while the application required around 76 MiB, triggered continuous Full GC, causing stop‑the‑world pauses, full‑line time‑outs, and a costly rollback.
Elastic CloudGCJVM
0 likes · 9 min read


ITPUBJan 19, 2022 · Databases
During nightly peaks, a social‑e‑commerce startup experienced hour‑long service outages due to MySQL timeouts; by analyzing traffic spikes, CPU usage, and slow‑SQL logs, the team identified un‑cached ranking queries and a 20‑minute cache refresh bottleneck, then implemented targeted caching, monitoring scripts, and fallback static pages to eliminate the issue.
CachingMySQLSlow SQL
0 likes · 14 min read

The article analyzes the Xi’an “Yima Tong” health‑code system outage, detailing the symptoms, root‑cause factors such as rate‑limiting gaps, server overload, architectural coupling, and ISP differences, and then offers short‑term, long‑term, design, high‑availability, and testing recommendations to prevent future crashes.
cloud-nativeincident-analysisperformance
0 likes · 13 min read

The article recounts a staged rollout of the Maybach service on elastic cloud, details the timeline of successful and failing deployments, analyzes JVM metrics revealing excessive Metaspace usage that triggered continuous full garbage collections, and explains how this caused system‑wide timeouts and a half‑hour outage.
Full GCJVMOperations
0 likes · 10 min read

The article analyzes Bilibili's recent half‑hour service disruption, explores technical rumors such as an etcd crash, examines Kubernetes‑based cloud‑native infrastructure, reviews similar historic outages, and offers expert recommendations for improving high‑availability and disaster‑recovery in large‑scale internet services.
BilibiliKubernetesOperations
0 likes · 8 min read

A sudden surge in live‑stream traffic overloaded the core streaming service because a push configuration mistakenly pointed to the same stream URL, creating a self‑referencing loop that repeatedly generated duplicate streams until the service capacity was exhausted.
Streaming Architecturebackend bugincident-analysis
0 likes · 4 min read

A new employee at an OTA company faced a mysterious outage where thousands of payment‑related messages piled up in the queue, the system auto‑recovered, and a detailed investigation revealed a stuck MySQL transaction caused by missing response timeout settings, leading to lock contention and message backlog.
HttpClientMessage Queueincident-analysis
0 likes · 7 min read

This article examines the hidden risks of hot‑reload mechanisms in web services, illustrates real incidents caused by careless configuration updates, analyzes root causes, and offers practical steps for detecting and fixing such pitfalls to improve operational reliability.
Configuration ManagementHot Reloadincident-analysis
0 likes · 7 min read


ITPUBMay 29, 2015 · Information Security
The article examines the Ctrip outage by analyzing observed symptoms, evaluating the likelihood of a large‑scale database attack versus node failures, and discussing how backup strategies and private‑cloud storage could affect data recovery in such a severe security incident.
Ctripbackup strategycloud storage
0 likes · 7 min read
