How Do Canal, Maxwell, Databus, and Alibaba DTS Enable Real-Time Data Replication?

Canal

Positioning: Based on database incremental log parsing, provides incremental data subscription & consumption, currently mainly supports MySQL.

Principle:

Canal simulates MySQL slave interaction protocol, disguises itself as a MySQL slave, and sends dump protocol to MySQL master.

MySQL master receives dump request and starts pushing binary log to the slave (i.e., Canal).

Canal parses the binary log object (originally a byte stream).

The overall parser process can be divided into several steps:

Connection obtains the last successfully parsed position (or initial position if first start).

Connection establishes connection and issues BINLOG_DUMP command.

MySQL starts pushing binary log.

Received binary log is parsed by Binlog parser, adding specific information.

Data is passed to EventSink module for storage, a blocking operation until successful.

After successful storage, the binary log position is periodically recorded.

Data filtering: supports wildcard patterns, table names, field content, etc.

Data routing/distribution: solves 1:n (one parser to multiple stores).

Data merging: solves n:1 (multiple parsers to one store).

Data processing: additional handling before storing, such as join.

Maxwell

Canal is Java‑based, with server and client components, many derivative applications, stable performance, and powerful features; it requires writing a client to consume parsed data.

Maxwell’s advantage over Canal is simplicity: it directly outputs data changes as JSON strings, eliminating the need to write a client.

Databus

Databus is a low‑latency change‑capture system that has become an essential part of LinkedIn’s data processing pipeline. It addresses the fundamental requirements of reliably capturing, streaming, and processing primary data changes.

Isolation between source and consumers.

Guarantees ordered, at‑least‑once delivery with high availability.

Supports consumption from any point in the change stream, including full data bootstrapping.

Partitioned consumption.

Source‑consistent storage.

Alibaba Cloud Data Transmission Service (DTS)

Data Transmission Service (DTS) is Alibaba Cloud’s data‑flow service that supports data interaction among RDBMS, NoSQL, OLAP and other sources. DTS provides data migration, real‑time data subscription, and real‑time data synchronization, enabling online migration, disaster‑recovery, active‑active deployment, cross‑region sync, real‑time data warehouses, query offloading, cache updates, and asynchronous notifications, helping build secure, scalable, highly available data architectures.

Advantages: DTS supports multiple data sources, offers rich transmission links with high performance and security, and provides many convenient features that simplify link creation and management.

Personal understanding: It works like a message queue that pushes packaged SQL objects, which you can parse with a custom service.

It eliminates costly deployment and maintenance. DTS is adapted for Alibaba Cloud RDS, DRDS, etc., handling binlog recycling, primary‑secondary switching, VPC network changes, and provides performance optimizations for stability, performance, and cost.