What Alibaba Cloud’s Epic Outage Reveals About Building Truly Resilient Systems

Preface

Last night Alibaba Cloud suffered an epic‑scale failure affecting Alibaba Cloud Drive, Taobao, Xianyu, DingTalk, Yuque and many other services, with topics like “Aliyun Drive crashed” and “Taobao crashed” trending on social media.

1. Yuque Anomaly

During the incident I was editing an article in Yuque and encountered a save error that caused the page to crash.

2. Social Buzz

My social feed instantly lit up with comments about the outage, underscoring its severity and wide impact.

3. Incident Timeline

Alibaba Cloud announced that at 17:44 on 12 Nov 2023, monitoring detected abnormal console access and API calls across cloud products, prompting engineers to intervene urgently.

At 18:54 the service began recovering in regions such as Hangzhou and Beijing, with other regions following suit.

The affected products included enterprise‑grade distributed application services, message queues, micro‑service engines, tracing, high‑availability services, real‑time monitoring, Prometheus, Kafka, machine learning, image search, and intelligent recommendation (AlRec).

Impacted regions spanned China (multiple data centers), Hong Kong, India, the United States (Silicon Valley and Virginia), Europe (London), Korea, Japan, the UAE, Singapore, Australia, Malaysia, the Philippines, Thailand, and more.

This is not Alibaba Cloud’s first large‑scale outage, and the exact cause remains unknown.

4. My Past Experience

Such incidents remind ordinary users that high‑traffic applications must be designed for high availability and multi‑region active‑active deployment.

When we built a gaming platform, we distributed login traffic across three data centers (Shenzhen 40%, Tianjin 30%, Chengdu 30%) and used multiple cloud providers to mitigate risks like power loss or natural disasters.

This active‑active architecture allowed traffic to be shifted instantly if one data center failed, minimizing user impact.