Step-by-Step Migration of Large MySQL Datasets to DBLE Using TiDB DM

Background

A customer has a MySQL instance with over a terabyte of data that needs to be migrated quickly and incrementally to DBLE; TiDB DM is chosen for its multi‑threaded export/import and binlog sync capabilities.

Migration Diagram and Server Environment

The migration topology includes a dm‑master, dm‑worker, source MySQL, DBLE, and DBLE datanodes with specific IPs, ports, and versions as listed in the table.

Install TiDB DM

On both dm‑master and dm‑worker nodes, create a tidb user, generate SSH keys, install tiup , and then install the dmctl component:

# useradd tidb
# echo "dmadmin" | passwd tidb --stdin
# ssh-keygen -t rsa
# ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected]
# curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh
# source .bash_profile
# which tiup
# tiup install dm dmctl:v2.0.7

Generate a topology template with tiup dm template > dm_topology.yaml , edit it to match the server roles, then deploy and start the cluster:

# tiup dm deploy dm-test v2.0.7 ./dm_topology.yaml --user tidb -i /home/tidb/.ssh/id_rsa
# tiup dm start dm-test

DBLE Configuration

Prepare db.xml , sharding.xml , and user.xml files to define database groups, sharding nodes, and users. Example snippets:

<dble:db ...>
  <dbGroup name="dbGroup1" ...>
    <dbInstance name="host_1" url="10.186.65.61:4408" user="sgy" password="..." maxCon="2000" minCon="50" primary="true" usingDecrypt="true"/>
  </dbGroup>
</dble:db>

Execute DBLE commands to create the physical databases:

# /data/mysql/base/5.7.36/bin/mysql -uroot -padmin -h 10.186.65.4 -P 9066
dble> create database @@shardingNode = 'dn$1-4';
dble> create database @@shardingNode = 'dm_meta';

Source MySQL Data Generation

Use sysbench to prepare and continuously insert data into the sbtest schema, generating millions of rows for sbtest1 and sbtest2 .

Start Data Synchronization Task

Create an encrypted password with tiup dmctl encrypt 'admin' , then define the source and task YAML files. Example source definition:

source-id: "mysql_source_14_4490"
from:
  host: "10.186.65.14"
  port: 4490
  user: "sgy"
  password: "dmXgktpuF18RP1mDN/B2UkA6pDN9PQ=="

Check the task configuration and start the task:

# tiup dmctl --master-addr 10.186.65.83:8261 check-task ./mysql_sync_to_dble.yaml
# tiup dmctl --master-addr 10.186.65.83:8261 start-task ./mysql_sync_to_dble.yaml

After starting, monitor the task status to see event counts, TPS, binlog positions, and any blocking DDLs.

Data Verification

Pause the sysbench load, then run SELECT COUNT(*) on both MySQL and DBLE to confirm row counts match. Perform an update on a source row and verify the change is reflected in DBLE.

Important Notes

TiDB DM version must be v2.0.7.

Disable DBLE slow‑query log during the full‑load phase to avoid excessive logging.

Ensure no pt‑kill or similar tools are running on the source MySQL, as they can interrupt the dump process.

If target table structures differ (e.g., partitions, charset), adjust them before or after migration.

Column order and types must match; mismatched varchar lengths can cause silent truncation.