Unlock Real-Time MySQL Binlog Streaming with Canal, Maxwell, and DTS

Canal

Positioning: based on database incremental log parsing, it provides incremental data subscription and consumption, currently mainly supporting MySQL.

Principle

Canal simulates the MySQL slave interaction protocol, masquerading as a slave and sending a dump request to the MySQL master.

The MySQL master receives the dump request and starts pushing binary logs to the slave (i.e., Canal).

Canal parses the binary log objects (originally a byte stream).

The overall parser process can be divided into several steps:

Connection obtains the last successfully parsed position (or an initial position if this is the first start).

Connection establishes the connection and issues the BINLOG_DUMP command.

MySQL begins pushing the binary log.

The received binary log is parsed by the Binlog parser, adding some specific information.

The parsed data is passed to the EventSink module for storage, which is a blocking operation until storage succeeds.

After successful storage, the binary log position is periodically recorded.

Data filtering: supports wildcard filter patterns, table names, field content, etc.

Data routing/distribution: solves 1:n (one parser to multiple stores) scenarios.

Data merging: solves n:1 (multiple parsers to one store) scenarios.

Data processing: additional handling before entering the store, such as joins.

Maxwell

Maxwell, also developed in Java, offers a simpler alternative to Canal. It directly outputs data changes as JSON strings, eliminating the need to write a custom client.

Databus

Databus is a low‑latency change capture system that has become an essential part of LinkedIn’s data processing pipeline. It addresses the fundamental requirements of reliable capture, flow, and processing of major data changes.

Isolation between source and consumers.

Guarantees ordered and at‑least‑once delivery with high availability.

Supports consumption from any point in the change stream, including full data bootstrapping.

Partitioned consumption.

Source consistency preservation.

Alibaba Cloud Data Transmission Service (DTS)

DTS is a data flow service provided by Alibaba Cloud that supports data interaction among various sources such as RDBMS, NoSQL, and OLAP.

It offers capabilities like data migration, real‑time data subscription, and real‑time data synchronization, enabling scenarios such as online migration, disaster recovery, multi‑active regions, cross‑border synchronization, real‑time data warehousing, query offloading, cache updates, and asynchronous notifications, helping build secure, scalable, and highly available data architectures.

Advantages

DTS supports multiple data sources and provides richer, higher‑performance, and more secure transmission links compared to third‑party tools. It also offers many convenient features that simplify the creation and management of transmission pipelines.

In essence, DTS acts like a message queue that pushes packaged SQL objects, which can be parsed by custom services.

It eliminates costly deployment and maintenance, offering adaptations for Alibaba Cloud RDS, DRDS, and handling binlog recycling, primary‑standby switching, and VPC network changes with high availability and performance optimizations.