How UCloud Executed a Seamless Hot Migration of Its Seoul Data Center

Plan

The project was divided into four phases: preparation, new‑site construction, migration, and decommissioning of the old site. The hot migration was likened to moving passengers from one high‑speed train to another while keeping both trains stationary, effectively merging the two data centers logically.

Challenges

The original Seoul facility used an outdated architecture, making core services such as ZooKeeper, the database middleware (udatabase), and the primary databases risky to move without causing large‑scale anomalies.

ZooKeeper Service Scaling

UCloud relies heavily on ZooKeeper for service registration and discovery. The old three‑node cluster was expanded to five nodes in the new site, then further to an eight‑node cluster (1 leader, 7 followers) that spanned both sites, allowing seamless data synchronization while keeping the old three nodes online until the final cut‑over.

Database Middleware udatabase Migration

Using the “same‑cluster expansion” mode (Figure 4), new udatabase instances were added with identical endpoint prefixes, allowing the existing client libraries to discover them via ZooKeeper and automatically balance traffic. When the old instances were shut down, all traffic shifted to the new Seoul nodes.

New‑Cluster Gray‑Scale Migration Mode

For services with higher risk (e.g., ULB, CNAT), a separate cluster was created (Figure 5). The new cluster used a different ZooKeeper path, and migration was performed via a dedicated platform (SRE‑Migrate), allowing instant rollback if issues arose.

Data Center Deployment Platform SRE‑Asteroid

The SRE‑Asteroid platform automated the deployment of nearly a hundred new clusters, handling configuration, version control, and rollout across the new Seoul facility.

Core Database Deployment and Migration

UCloud's MySQL clusters (1 primary, 1 HA replica, 2 read‑only replicas, 1 backup) were duplicated in the new site. The new primary acted as a replica of the old primary, using asynchronous binlog replication. During a low‑traffic window, the console services were stopped, the udatabase VIPs were switched to the new MySQL cluster, and the old primary was set to read‑only. Consistency was verified with pt‑table‑checksum.

Wrap‑up

After all services were migrated and verified, traffic monitoring confirmed no remaining requests to the old site. The old three‑node ZooKeeper cluster was finally decommissioned, leaving the new five‑node cluster fully operational. The entire hot migration was completed in five months with minimal user impact.

— END —