Seamless Migration of Xianyu Product Database Using TDDL and DTS

Background: Rapid system iteration often leads to core and non‑core data sharing the same physical database, causing stability risks such as slow SQL, disk I/O contention, and overall performance degradation. To isolate core business tables, a migration to a dedicated database is required, with the key challenge of achieving a smooth, user‑transparent data transfer.

Current State: Xianyu’s product data exceeds tens of billions of rows and is served by a MySQL cluster employing sharding, read/write splitting, and the TDDL middleware for unified management.

Sharding Principle: Horizontal partitioning splits a single logical database into multiple physical instances, alleviating single‑node bottlenecks. However, it introduces complexity in schema changes and management.

Global Sequence Generation: TDDL provides a sequence mechanism that allocates ID blocks in memory after a database update, offering high efficiency but not guaranteeing strict auto‑increment order. During migration, a larger primary‑key offset is set to avoid conflicts.

Migration Strategy: The migration consists of system refactoring, stability assurance, and data migration phases. System refactoring includes dual‑write switches, loading two TDDL data sources, and creating new sequences. Stability assurance involves stress testing, online traffic replay, and fault‑injection drills.

Data Migration: Utilizes Alibaba Cloud DTS for full‑load, incremental sync, consistency checks, and reverse tasks. Full‑load copies historical data based on a chosen timestamp. Incremental sync captures binlog changes after the full‑load point, with a small time buffer to ensure eventual consistency.

Consistency Verification: After full and incremental migration, batch queries compare source and target tables to detect any discrepancies before go‑live.

Rollback Plan: The primary principle is no data loss. If the new database fails during dual‑write, the write switch is turned off immediately. After cut‑over, if issues arise, traffic can be reverted to the old database while reverse sync maintains consistency.

Conclusion: By designing a detailed migration plan and leveraging DTS, Xianyu successfully migrated its massive product database, achieving physical isolation and improved stability. The approach can serve as a reference for similar large‑scale database migrations.