Automated Migration Platform for Qunar DBA: Architecture, Implementation, and Outcomes

1. Overview

The article introduces Qunar DBA's automated migration program, which automates common DBA operations such as out‑of‑warranty machine replacement, cluster architecture upgrades, and database version upgrades, providing configurable workflows to reduce manual errors and improve efficiency.

2. Background

DBA work is heavy and repetitive, involving backups, monitoring, cluster scaling, data migration, and performance tuning, while also handling frequent developer requests, leading to high training costs for newcomers and a high risk of human‑induced failures during complex migration procedures.

3. Automated Migration Implementation

Automation replaces manual steps; the first effort focused on cluster installation and deployment, which involved complex components such as master‑slave semi‑sync, PXC clusters, Sentinel clusters, configuration centers, and Zookeeper. Each deployment is defined as a series of tasks and executed asynchronously via a platform‑added job.

During development, challenges arose such as long migration cycles requiring code changes and high‑risk operations (master‑slave switch, cut‑over) that needed careful scheduling, prompting a design that splits complex processes into independent sub‑tasks that can be paused, resumed, and orchestrated.

3.1 Overall Architecture

3.1.1 Task Pool

Each independent task represents a specific DBA operation, ranging from a single command (e.g., start/stop database) to a series of actions; finer granularity improves reuse. Remote execution is performed via SaltStack.

3.1.2 Task Template

Backend‑defined templates combine common sub‑tasks into ready‑to‑run migration flows such as cluster version upgrades, architecture upgrades, out‑of‑warranty replacements, deployments, and uninstallations.

Statistics of usage include:

MMM version upgrades saved ~1 hour per cluster.

Cluster deployment reduced from 30 minutes manually to ~3 minutes automatically, with >180 deployments.

Out‑of‑warranty machine replacement processed >400 machines in 4 years, saving >90% of time.

Single‑instance backup migration for resource consolidation.

Cluster uninstallation handling complex components (ZK, Sentinel, Git) with one‑click automation.

MMM to PXC architecture upgrade achieved >60% coverage, cutting upgrade time from 3 hours to 20 minutes with zero‑downtime.

3.1.3 Migration Task

Migration tasks are configured via a web UI, showing logs, snapshots (parameters), workflow, and management controls. The management page supports pause, manual scheduling, and marking tasks as completed.

3.1.4 Scheduled / Manual Scheduling

Scheduled jobs poll the task table every 10 seconds, enqueueing eligible tasks; manual scheduling is triggered by a button click that calls the dispatch API.

3.1.5 Execution Queue

The system uses Celery as a task queue, enabling multithreaded concurrent execution of multiple workflows; results are fed back into the corresponding task steps.

3.2 Tools

Development tools: Python 3.6, Tornado, Celery; Remote command execution: SaltStack.

4. Conclusion

The automated migration program decomposes lengthy, complex migration procedures into composable tasks, allowing DBAs to configure and launch migrations without direct server access, dramatically improving operational efficiency and minimizing human error. Qunar DBA has already automated over 60% of MMM‑to‑PXC upgrades, laying groundwork for future resource‑pool management and instance migration.