Alibaba's Massive Data Architecture: Sharding, Multi‑Data‑Center Synchronization, and Caching Strategies

Author Tao Yong, who joined Alibaba in 2005 and later led the distributed database team, explains the challenges of massive data growth across Alibaba's multiple sites (Chinese, international, Japanese) where core tables reach billions of rows, causing capacity, performance, and distribution issues.

Initially relying on Oracle, Alibaba recognized bottlenecks around 2007‑2008 and began splitting databases. Horizontal sharding moved core tables to MySQL clusters, alleviating CPU and I/O pressure while improving TPS and capacity.

The strategy involves two main actions: distributing load to transform a centralized architecture into a distributed one, and using varied storage solutions (Oracle, MySQL, KV stores, NoSQL) based on data characteristics.

Horizontal sharding is the primary method for B2B, moving large tables from Oracle to MySQL; vertical sharding is used as a transitional step. Multiple storage options are selected per business need, with Oracle for core data, MySQL for high‑volume tables, and KV stores for less relational data.

Three Alibaba products address these challenges: Erosa parses MySQL bin‑log in real time; Eromanga publishes the parsed changes for subscription by downstream systems; Otter synchronizes data across IDC sites, providing bidirectional, transaction‑level replication similar to SharePlex.

Data‑sync conflicts are handled by prioritizing one site’s data over another, with future plans for more sophisticated conflict‑resolution policies.

Caching is implemented in three layers—from front‑end image and page caches to back‑end local and remote (distributed) caches. Alibaba uses various cache engines (Berkeley DB, Memcached) and a unified front‑end wrapper to maximize hit rates and manage cache lifecycles.

The “拆迁大队” (demolition team) metaphor describes the team responsible for breaking large Oracle tables into MySQL shards using strategies such as field‑level, table‑level, and schema‑level splitting, often orchestrated through the proprietary MySQL proxy Cobar, which routes queries based on custom sharding rules.

Data migration follows a strict five‑step process, gradually moving reads/writes from Oracle to MySQL while ensuring minimal impact on users, typically performed during short maintenance windows.

Backend data warehousing (DW) leverages Oracle RAC, Greenplum, and Cobar for both offline and near‑real‑time analytics, with recent improvements reducing data latency from a day to five minutes thanks to Erosa.

Overall, Alibaba aims to build a unified platform that serves both OLTP and OLAP workloads, combining distributed database techniques, high‑performance sharding, and robust synchronization to meet the demands of massive, globally distributed traffic.