How Alibaba Cloud’s Network Evolved from Classic to Intelligent VPC Architecture

1. Business Demand Drives Network Change

As large enterprises migrate to the cloud, richer industry scenarios and diverse services on Alibaba Cloud raise stringent requirements for network scale, performance, and elasticity, prompting continuous optimization from classic to dedicated networks, control plane 1.0 to 3.0, and the adoption of intelligent NICs.

2. From Classic Network to Dedicated VPC

Early Alibaba Cloud users, mainly from the Internet sector, needed self‑planned, securely isolated networks, leading to the evolution from a large‑layer classic network to tenant‑controlled VPCs.

Classic networks suffered from:

Insufficient security isolation due to large‑layer design.

Strong coupling with physical switches, limiting flexibility.

Limited address space causing IP exhaustion.

Restricted VM migration domains, hindering rapid failover.

Inability for customers to plan their own IP schemes.

These drawbacks drove the launch of Alibaba Cloud VPC in 2014, an overlay network built on VXLAN that provides logical isolation between VPCs and decouples from the physical layer.

From 2014 to 2018, Alibaba Cloud promoted Classic‑to‑VPC migration, introduced ClassicLink in 2017 to enable temporary inter‑connectivity, and made VPC the default network for new users in 2016.

3. Business Model Drives VPC Underlying Evolution

The VPC control plane consists of a Controller, a virtual Gateway, and virtual switches (VSW). Their evolution paths differ due to distinct roles and traffic volumes.

The VPC controller processes user requests from the console and orchestrates the data‑plane components.

Challenges for the control plane in massive cloud networks include:

Much larger routing tables (millions of entries) compared to traditional routers.

Wider distribution of flow tables across many physical hosts.

Fast propagation of configuration changes (sub‑200 ms RTO).

Controller 1.0 handled simple, low‑scale workloads synchronously. Controller 2.0 introduced an asynchronous middle‑layer to improve latency. Controller 3.0, launched for the million‑VM VPC era, splits into four layers (API, orchestration, task, and configuration delivery) and separates VSW and Gateway engines for horizontal scaling.

4. Internal Service De‑gateway: Traffic Shifting

In data‑center networks, north‑south traffic (external ↔ internal) and east‑west traffic (internal ↔ internal) are distinguished. In Alibaba Cloud's VPC, all traffic that must pass through a gateway is north‑south, while direct VM‑to‑VM traffic becomes east‑west.

To bypass the centralized gateway bottleneck, Alibaba Cloud pushes VM‑to‑VM traffic down to the virtual switch layer, using a Route Synchronization Protocol (RSP) that synchronizes VM‑to‑physical‑host routes via Remote Control Messages (RCM).

5. Boundary Gateway Hardwareization

Initially, Alibaba Cloud gateways comprised IGW (Internet), VGW (vRouter), and CGW (Customer) on x86 servers using DPDK. To reduce cost and improve performance, IGW, VGW, and CGW were merged into a unified XGW (Any) gateway, boosting CPU cores from 16 to 32 and bandwidth from 40 G to 160 G.

Programmable 3.2 T switching chip, 32 × 100 GE QSFP28 ports.

Up to 2 CPUs, 26 cores each, 128 GB DRAM.

6 × PCIe slots with FPGA expansion.

Hardware gateways eliminated the single‑core PPS bottleneck of DPDK x86 clusters, reducing the required number of servers from dozens to a single appliance for 1.6 T traffic.

6. Embracing Smart NICs

Virtual switches (AVS) originally ran on host CPUs, consuming resources and incurring copy overheads. Smart NICs (e.g., Alibaba Cloud MOC cards) offload AVS functions to on‑board CPUs and DMA engines, freeing host CPU cycles and improving bandwidth.

The latest MOC 2.5 supports 200 G bandwidth, 5 million pps, and features such as traffic mirroring, eRDMA, VPC encryption, and jumbo frames.

7. Business Element Virtualization (NFV)

Traditional network appliances are costly and rigid. NFV decouples network functions from dedicated hardware, deploying them as virtualized services (NAT, SLB, CEN, VPN) on standard x86 servers.

The Alibaba Cloud NFV platform follows ETSI MANO, comprising VIM (resource management), VNFM (function lifecycle), and NFVO (orchestration). Fastpath ECS handles stateless forwarding, while Slowpath ECS processes offloaded flow rules, enabling scalable, elastic network services.

8. Summary

Classic network limitations (security, coupling, address space) drove the shift to VPC.

VPC controller evolution targets massive scale and extreme elasticity.

Internal service de‑gateway shifts VM‑to‑VM traffic east‑west, alleviating centralized gateway bottlenecks.

20 % of customers generate 80 % of traffic, prompting hardware gateway adoption.

Smart NICs break the performance ceiling for single VMs.

NFV virtualizes network elements, delivering elasticity and cost savings.

NFV platform’s fast‑slow path separation simplifies network function development.

Appendix

The Alibaba Cloud VPC team, in collaboration with Zhejiang University, contributed the Achelous paper to SIGCOMM, following Azure’s VFP (NSDI ’17) and GCP’s Andromeda (NSDI ’19) as the third top‑conference publication on public‑cloud network foundations.

For deeper technical details, see the paper: https://dl.acm.org/doi/10.1145/3603269.3604859