Master Database Migration to Cloud: Challenges & Solutions with Baidu DTS

1 Database Migration Challenges

According to the 2023 Database Development Research Report, China's database market grew from 40 billion RMB last year to an expected 54 billion RMB this year and may reach 128 billion RMB by 2027, with a compound annual growth rate of 26%.

The market is shifting toward public‑cloud databases, which now account for about 60% of the total and grow at roughly 50% annually, far outpacing on‑premises deployments.

Key technical challenges of cloud migration include:

Choosing the right cloud engine, architecture, and pricing plan while ensuring compatibility and business‑specific tuning.

A lengthy migration workflow that must handle network bridging, full data transfer, incremental sync, and final cut‑over.

Maintaining efficiency and disaster‑recovery capabilities throughout the process.

Guaranteeing data consistency, as any loss or duplication can severely impact online services.

2 Baidu Intelligent Cloud DTS Solution

Baidu emphasizes that smoothness (usability, compatibility, minimal business impact) and reliability (consistency, rollback, high availability) are core customer requirements.

2.1 Smooth Migration

Usability: out‑of‑the‑box, fully managed migration workflow.

Compatibility: supports heterogeneous source and target engines, architectures, versions, and network environments.

Business impact: enables nonstop migration of accounts, structures, full and incremental data.

2.2 Reliable Migration

Consistency: ensures source and target data are identical and provides verification tools.

Rollback: one‑click reverse sync restores cloud‑down environments for disaster recovery.

High availability: checkpointing, real‑time disaster recovery, and automatic failover during primary‑standby switches.

2.3 Migration Process

DTS divides migration into three phases: pre‑migration (assessment and selection), migration (full‑load, incremental sync, cut‑over), and post‑migration (intelligent dashboard for monitoring and tuning).

Pre‑migration evaluates engine choices, performance, and network connectivity (public network, dedicated line, VPN, or cloud‑native options). Migration moves accounts/roles, schema, full data, and incremental changes without downtime, then validates consistency before traffic cut‑over. After cut‑over, DTS can reverse‑sync traffic back to the on‑premise environment if needed.

2.4 Data Transfer Architecture

The data plane follows an ETL plug‑in design:

Extract (E) : supports full‑load, incremental CDC, and parallel table sharding.

Transform (T) : normalizes source data to a unified abstract format, applies filtering, mapping, and protocol conversion.

Load (L) : parallel batch loading with table‑ or primary‑key‑level concurrency, preserving order for incremental streams.

Consistency verification uses a two‑round recheck to reduce false positives caused by millisecond‑level CDC lag.

2.5 Exactly‑Once Guarantees

At‑Least‑Once guarantees no data loss via monotonic version tracking. Exactly‑Once is achieved by leveraging target‑side unique constraints, UUID tagging for message queues, or command rewriting for key‑value stores.

2.6 High Availability & Performance

Checkpoint‑based resume ensures no data loss across restarts.

Real‑time disaster recovery with automatic split‑brain detection and self‑healing.

Switch‑over self‑healing automatically discovers new nodes and resumes transfer.

High throughput: parallel full‑load, table/primary‑key parallel conversion, batch INSERT merging, and network latency amortization.

Low latency: CDC‑based incremental capture, streaming pipeline, message‑queue buffering, and logical transaction merging.

MySQL benchmarks show up to 200 k rows/s full‑load and 10 k rows/s incremental with millisecond‑level delay.

3 Use Cases

Three real‑world deployments illustrate DTS capabilities:

A large video‑streaming platform migrated over 1.5 W clusters (MySQL, Redis, MongoDB) with nonstop migration, self‑healing, and sub‑4‑day per‑cluster cycles.

A major commercial bank achieved zero‑RPO, sub‑30‑second RTO disaster recovery for 64‑shard workloads with 500 k+ TPS and minute‑level sync latency.

A large language‑model service realized second‑level sync for model traces and real‑time analytics, supporting 470+ continuous tasks.

DTS supports eight typical scenarios: nonstop cloud migration, active‑active, multi‑cloud DR, event‑driven, migration for domestic‑innovation platforms, cache updates, real‑time analytics, and real‑time lake/warehouse ingestion.