Migrating a Multi-Cloud Cluster in 2 Hours: Key Strategies and Lessons

Background

Cloud‑native brings engineering efficiency, but differing cloud provider architectures make whole‑cluster migration complex. KuJiaLe performed a full cluster cutover from Cloud A to Cloud B in early 2022 and shares the experience for quality teams.

Goals and Constraints

Ensure the migration does not affect important merchants.

Achieve a single successful cutover because rollback is difficult.

Limit downtime to two hours to meet international business requirements.

Set a clear deadline; the cutover was scheduled during the Chinese New Year low‑traffic period.

Testing Strategy

Test Objects

Code adaptations for middleware changes.

Middleware configuration differences.

ZooKeeper and other configuration changes.

Network topology after multi‑cloud deployment.

Domain name changes.

Cluster Preparation

A simulated “mirrored” cluster was built in Cloud B, isolated from production, with data and configuration fully synced from Cloud A. Access was provided via VPN or dedicated VMs, and the environment could be reset repeatedly.

Multi‑Round Test Plan

Original Cloud A environment testing.

Beta “mirrored” environment smoke test in Cloud B.

Production‑grade testing in Cloud B.

Mirror traffic replay using nginx mirror.

Performance stress testing with goreplay.

Internal beta testing (bug‑bash).

Gradual gray‑release testing.

Rollback verification before go‑live.

Final acceptance testing after data sync.

Full production validation.

Key Practices

Traffic Replay

Used nginx mirror to replay live traffic to the simulated cluster, uncovering functional gaps and establishing performance baselines.

Performance Testing

A three‑stage performance testing process identified more than 20 issues, including storage configuration problems and missing optimizations.

Latency Simulation

Emulated cross‑cloud latency with

tc qdisc add dev eth0 root netem delay 100ms 10ms

, revealing unacceptable response times for some interfaces.

Project Communication

Adopted hierarchical communication, online forms, and dashboards to reduce coordination overhead among 50+ test teams and 200+ developers.

Conclusion

The two‑hour cutover was high‑risk but successful thanks to thorough preparation, multi‑stage testing, and proactive communication. Test PMs play a critical role in identifying risks and ensuring quality in large‑scale migrations.