Automating Test Environment Data Synchronization with Alibaba DataX and QDataManager

Before the solution, baseline data for test environments was maintained manually by executing SQL statements on each environment's database, leading to inefficiencies and inconsistencies when data was modified and not restored.

The team introduced a two‑stage approach: first, one‑click deployment handled configuration, database installation, and service deployment; second, they addressed baseline data synchronization by leveraging Alibaba's open‑source DataX framework and building a task management interface called QDataManager.

DataX provides heterogeneous data synchronization, easy extensibility through Reader and Writer plugins, high efficiency with parallel task execution, strong reliability with data type handling and monitoring, speed control, and fault‑tolerance via retry strategies.

Using DataX, the QDataManager allows users to add, edit, query, and execute synchronization tasks either manually or on a schedule. The system generates the required DataX JSON configuration for each task and stores task metadata in a management database, enabling a single source of truth for baseline data.

Example configuration for synchronizing the subject_info table in the Z environment:

{
	"job": {
		"content": [{
			"reader": {
				"name": "mysqlreader",
				"parameter": {
					"column": ["*"],
					"connection": [{
						"jdbcUrl": ["jdbc:mysql://ip:port/pay_accounting"],
						"table": ["subject_info"]
					}],
					"password": "passw",
					"username": "user",
					"where": ""
				}
			},
			"writer": {
				"name": "mysqlwriter",
				"parameter": {
					"column": ["*"],
					"connection": [{
						"jdbcUrl": "jdbc:mysql://ip:port/pay_accounting",
						"table": ["subject_info"]
					}],
					"password": "passw",
					"preSql": ["truncate table subject_info"],
					"session": ["set session sql_mode='ANSI'"],
					"username": "user",
					"writeMode": "insert"
				}
			}
		}],
		"setting": {
			"speed": {
				"channel": "1"
			}
		}
	}
}

The configuration explains the reader and writer nodes, column selection, JDBC URLs, table names, optional where clause for incremental sync, pre‑SQL statements, session settings, and write mode (insert or replace).

Since deployment, the payment center runs daily scheduled synchronizations, freeing engineers from manual data updates, reducing debugging time caused by inconsistent test data, and overall improving test efficiency.