Alibaba Cloud Data Integration (DataX) Architecture, Design Principles, and Solution Overview

The talk introduces Alibaba Cloud Data Integration, a commercial product built on the open‑source DataX engine, aimed at providing high‑speed, stable data movement across heterogeneous sources in complex network environments and supporting diverse data synchronization scenarios.

Four primary data integration scenarios are described: migrating on‑premise databases to the cloud, building real‑time data warehouses, platform data fusion across cloud services, and disaster‑recovery backup for hot and cold data.

The product’s positioning emphasizes offline and real‑time coverage, network‑agnostic solutions, security isolation between development and production, extensive source/target support (50+ offline, 10+ real‑time), ready‑made solutions for common ETL tasks, and a comprehensive monitoring and alerting system.

DataX’s core design abstracts data sources and sinks into Reader and Writer plugins with a unified interface, enabling linear data flow and easy extensibility; the open‑source repository is https://github.com/alibaba/DataX.

Two synchronization mechanisms are explained: offline jobs run as DataX processes using JDBC or SDKs to read data and write to targets via a producer‑consumer model, while real‑time sync captures change logs (Binlog, CDC) or message streams, processes events, and writes to destinations, leveraging a checkpoint system similar to Flink’s barrier‑based snapshots for fault‑tolerant streaming.

Deployment modes include Standalone (single‑process), Distributed (task groups across multiple workers for linear scalability), and On‑Hadoop (tasks executed as MapReduce jobs on an existing Hadoop/YARN cluster).

Solution systems are presented for offline full‑library migration, real‑time full‑incremental sync, and intelligent real‑time data warehousing, illustrating how DataX integrates with MaxCompute, Hologres, ElasticSearch, Kafka, and other Alibaba Cloud services to support end‑to‑end data pipelines.

Finally, the relationship between DataWorks and Data Integration is outlined: Data Integration serves as a core module within DataWorks, providing unified data movement capabilities that feed into metadata management, task scheduling, data development, and data services, while also being usable independently via APIs.