Unlock Real-Time MySQL Binlog Streaming with Canal, Maxwell, and DTS

Canal

Positioning: based on database incremental log parsing, it provides incremental data subscription and consumption, currently supporting MySQL.

Principle:

Canal simulates the MySQL slave interaction protocol, masquerading as a MySQL slave and sending a dump request to the MySQL master.

The MySQL master receives the dump request and starts pushing binary logs to the slave (i.e., Canal).

Canal parses the binary log objects (originally a byte stream).

The overall parser process can be divided into several steps:

Connection obtains the last successfully parsed position (or an initial position if this is the first start).

Connection establishes a connection and issues the BINLOG_DUMP command.

MySQL begins pushing the binary log.

The received binary log is parsed by the Binlog parser, adding some specific information.

The parsed events are passed to the EventSink module for storage, which is a blocking operation until storage succeeds.

After successful storage, the binary log position is periodically recorded.

Data filtering: supports wildcard patterns for table names, field contents, etc.

Data routing/distribution: solves 1:n (one parser to multiple stores) scenarios.

Data merging: solves n:1 (multiple parsers to one store) scenarios.

Data processing: performs additional handling such as joins before data is stored.

Maxwell

Canal is written in Java, consisting of a server and client, with many derived applications; it is stable and feature‑rich, but requires developers to write a client to consume the parsed data.

Maxwell’s advantage is its simplicity: it directly outputs data changes as JSON strings, eliminating the need to write a custom client.

Databus

Databus is a low‑latency change‑capture system that has become an essential part of LinkedIn’s data processing pipeline. It addresses the fundamental requirements of reliable capture, flow, and processing of major data changes. Databus provides the following capabilities:

Isolation between source and consumers.

Guarantees ordered, at‑least‑once delivery with high availability.

Ability to consume from any point in the change stream, including full bootstrap of the entire dataset.

Partitioned consumption.

Source‑consistent persistence.

Alibaba Cloud Data Transmission Service (DTS)

Data Transmission Service (DTS) is an Alibaba Cloud offering that supports data interaction among various sources such as RDBMS, NoSQL, and OLAP. DTS provides capabilities including data migration, real‑time data subscription, and real‑time synchronization, enabling scenarios like non‑stop migration, disaster recovery, multi‑active deployments, cross‑border synchronization, real‑time data warehouses, query/report offloading, cache updates, and asynchronous notifications, helping build highly secure, scalable, and highly available data architectures.

Advantages: DTS supports many data sources and offers a rich set of transmission methods with higher performance, stronger security, and numerous convenience features that simplify the creation and management of transmission pipelines.

Personal interpretation: DTS functions like a message queue that pushes packaged SQL objects; developers can build services to parse these SQL objects.

It eliminates costly deployment and maintenance overhead. DTS is specially optimized for Alibaba Cloud RDS, DRDS, handling binlog recycling, master‑slave switchover, VPC network changes, and providing high‑availability subscription. For stability, performance, and cost reasons, its use is recommended.