How Alipay Handles 540k TPS: Inside the LDC Architecture, Unitization and CAP Analysis

LDC and Unitization Overview

Since the first Double‑11 in 2008, Ant Financial's payment peak grew from 20,000 transactions per minute to 540,000 TPS in 2019. The core breakthrough is the Logical Data Center (LDC), a unit‑based design that splits data across multiple logical zones rather than relying on a single physical IDC.

Unitization Design

LDC divides the system into three logical unit types:

RZone (Region Zone) : the smallest deployable unit that holds a partitioned set of user data. Each RZone runs its own application servers and a dedicated database instance, providing isolation and linear scalability.

GZone (Global Zone) : hosts globally shared, non‑shardable data such as system configuration. Only one copy exists, and it is typically deployed for disaster‑recovery across regions.

CZone (City Zone) : a hybrid zone that stores data with a "write‑read time lag" pattern. Writes go to GZone and are asynchronously replicated; reads are served locally, reducing cross‑region latency.

Data is classified into two main categories:

"User‑flow" data (orders, logs, behavior) – naturally isolated per user and best suited for RZone deployment.

"Shared" data – further split into user‑shared (accounts, messages) and system‑wide (products, configs). The former lives in GZone, the latter may be placed in CZone when the write‑read lag permits.

Traffic Routing and Disaster Recovery

Incoming requests are first routed by a custom Global Server Load Balancer (GLSB) that resolves the client IP to the nearest IDC domain (e.g., cashieridc‑1.alipay.com). The request reaches the IDC’s Spanner reverse‑proxy, which reads the routing table to determine the target RZone. If the target RZone resides in another IDC, the request is forwarded across data centers.

During a disaster, traffic is cut over in two steps:

Reassign the data‑partition access rights from the failed RZones to healthy ones (e.g., move partitions a and b from RZ0/RZ1 to RZ2/RZ3).

Update the user‑ID‑to‑RZone mapping so subsequent requests are directed to the new RZones.

Because database mapping is changed before traffic, no requests are lost to an unavailable partition.

CAP Theorem Overview

CAP states that a distributed system can simultaneously guarantee at most two of Consistency, Availability, and Partition tolerance. The article reviews the definitions of Consistency (C), Availability (A), and Partition tolerance (P) and explains why real‑world systems must make trade‑offs.

CAP Analysis of Alipay’s Architecture

At the application layer, horizontal scaling of stateless services already provides Availability and Partition tolerance (AP), but consistency is delegated to the database layer.

Traditional master‑slave MySQL clusters achieve Consistency and Availability (CA) only when no network partition occurs; a partition leads to split‑brain and loss of consistency.

Alipay’s LDC relies on OceanBase (OB), a distributed database that implements the Paxos consensus algorithm. OB requires writes to be acknowledged by a quorum of (N/2)+1 nodes, ensuring that even if a minority of nodes are partitioned, the system remains available (AP) and eventually consistent.

During a partition, OB may sacrifice immediate consistency, but the Paxos quorum guarantees that only one value can be committed, and the remaining nodes converge to that value once the partition heals, achieving eventual consistency.

Database‑Level CAP Details

Partition tolerance: OB nodes communicate and can tolerate the loss of a minority of replicas.

Availability under partition: Transactions succeed as long as a quorum is reachable; the system remains operational.

Consistency under partition: Immediate strong consistency is not guaranteed, but Paxos ensures a single committed value and eventual convergence.

Thus, OceanBase, and by extension Alipay’s LDC, fall into the AP category with eventual consistency (C).

Key Takeaways

Unitization (RZone) based on user‑sharding enables linear TPS growth.

GZone stores truly global data; CZone optimizes "write‑read time lag" data for local reads.

Traffic routing via GLSB and Spanner allows dynamic redirection and seamless cross‑IDC failover.

OceanBase’s Paxos‑based consensus provides partition tolerance and high availability while delivering eventual consistency.

The combination of logical unitization and a consensus‑driven database lets Alipay sustain >540k TPS and scale further.

References

Practice of Cloud System Administration, Volume 2 – Limoncelli, Chalup, Hogan.

MySQL 5.7 Semi‑synchronous Replication – https://www.cnblogs.com/zero-gg/p/9057092.html

BASE Theory – https://www.jianshu.com/p/f6157118e54b

Keepalived – https://baike.baidu.com/item/Keepalived/10346758

Paxos – https://en.wikipedia.org/wiki/Paxos_(computer_science)

OceanBase Technical Overview – https://www.cnblogs.com/antfin/articles/10299396.html

RZ0* --> a
RZ1* --> b
RZ2* --> c
RZ3* --> d

[00-24] --> RZ0A(50%), RZ0B(50%)
[25-49] --> RZ1A(50%), RZ1B(50%)
[50-74] --> RZ2A(50%), RZ2B(50%)
[75-99] --> RZ3A(50%), RZ3B(50%)