Flink CDC vs Canal: Building 99.99% Consistent Real-Time Data Pipelines

Flink CDC: The Last Mile for Real-Time Data Lake Ingestion

Real-time data warehouses and data lake architectures are replacing traditional T+1 offline warehouses. The key challenge is capturing database changes (Insert/Update/Delete) with low latency, high reliability, and zero intrusion. CDC (Change Data Capture) splits into query-based CDC and binlog-based CDC .

Legacy Architecture vs. Flink CDC

The old pipeline required three stages:

Enable MySQL binlog

Canal syncs binlog to Kafka

Flink consumes Kafka for business processing

This long chain introduces operational complexity. Flink CDC eliminates Canal and Kafka by reading binlog directly from the database, shortening the path and reducing components.

Core Value of Flink CDC

Non-intrusive: No business code changes or triggers; reads database logs only.

End-to-end Exactly-Once: Leverages Flink Checkpoint mechanism for no-loss, no-duplication guarantees.

Unified processing model: CDC data enters Flink as streams, enabling windowing, dimension-table joins, and state management.

Lakehouse bridge: Connects OLTP systems to data lakes (Iceberg, Delta Lake, Hudi) for real-time lakehouse ingestion.

Core Principles

Flink CDC wraps Debezium's Source Connector into Flink's SourceFunction. Workflow:

Full snapshot at startup

Switch to incremental log (Binlog/Redo Log)

Unified event format: All records (snapshot + incremental) output as RowData or JSON with operation type (INSERT/UPDATE/DELETE), timestamp, before/after images.

Checkpoint consistency: Flink checkpoints align source offsets with sink state.

Note: Flink CDC 2.0+ introduces lock-free snapshot and parallel reading , dramatically improving large-table initialization performance.

MySQL Integration via DataStream API

Prerequisites:

Enable binlog: binlog_format=ROW, binlog_row_image=FULL User needs REPLICATION SLAVE, REPLICATION CLIENT, SELECT privileges

Core code:

public static void main(String[] args) throws Exception {

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    env.setParallelism(1);
    Properties properties = new Properties();
    properties.setProperty("bootstrap.servers", "localhost:9092");
    properties.setProperty("group.id", "test-group");

    JdbcSource<RowData> source = JdbcSource.<RowData>builder()
            .setDrivername("com.mysql.jdbc.Driver")
            .setDBUrl("jdbc:mysql://localhost:3306/test_db")
            .setUsername("flink_cdc_user")
            .setPassword("password")
            .setQuery("SELECT id, name, age, email FROM test_table")
            .setRowTypeInfo(Types.ROW(Types.INT, Types.STRING, Types.INT, Types.STRING))
            .setFetchSize(1000)
            .build();

    DataStream<RowData> stream = env.addSource(source);

Run command example:

$ bin/flink run -c com.example.MyCDCJob ./my-cdc-job.jar --database.server=mysql.example.com --database.port=3306 --database.name=mydb --database.username=myuser --database.password=mypassword --table.name=mytable --debezium.plugin.name=mysql --debezium.plugin.property.version=1.3.1.Final

Output shows change events with table name, primary key, and before/after values. For instance, a record's age field changes from 25 to 27:

[INFO] Change data for table: mytable.
[INFO] Record key: {"id": 1}, record value: {"id": 1, "name": "Alice", "age": 25}.
[INFO] Record key: {"id": 1}, record value: {"id": 1, "name": "Alice", "age": 27}.

MySQL Integration via Flink SQL

Create a CDC source table and query:

-- Create MySQL CDC source table
CREATE TABLE mysql_users (
  id INT PRIMARY KEY NOT ENFORCED,
  name STRING,
  email STRING,
  update_time TIMESTAMP(3)
) WITH (
  'connector' = 'mysql-cdc',
  'hostname' = 'localhost',
  'port' = '3306',
  'username' = 'flinkuser',
  'password' = 'flinkpw',
  'database-name' = 'test_db',
  'table-name' = 'users'
);

-- Query and output
SELECT * FROM mysql_users;

High-Frequency Interview Questions

Q1: Flink CDC vs. Canal / Maxwell

Integration: Flink CDC deeply integrates with Flink for direct stream computing; Canal/Maxwell run as standalone services requiring extra Flink integration.

Semantics: Flink CDC natively supports Checkpoint and Exactly-Once; Canal requires custom offset management.

Full + Incremental: Flink CDC automatically switches between snapshot and incremental; Canal only supports incremental.

Q2: How Does Lock-Free Snapshot Work?

Flink CDC 2.0 introduces Chunk-based Snapshot:

Split table by primary key into chunks

Each chunk reads independently, recording high/low watermarks

Concurrent writes allowed during read; binlog compensates intermediate changes

Merge snapshot and incremental for final consistency

Q3: Handling DDL Changes (e.g., Add Column)

Current limitation: Flink CDC does not support dynamic DDL sync by default (errors or ignores).

Solutions:

Manual job restart (suitable for low-frequency changes)

Schema Registry + dynamic deserialization (e.g., Avro)

Flink 1.17+ Dynamic Table Options for schema evolution (experimental)

Q4: Can Flink CDC Capture DELETE Operations?

Yes. DELETE events output with op='d' and include the full before-image row (requires database log to contain before image, e.g., MySQL ROW format).

Q5: Optimizing Large-Table CDC Performance

Upgrade to Flink CDC 2.3+, enable parallelism parameter

Increase source parallelism (requires even primary-key distribution)

Tune checkpoint interval (avoid overly frequent checkpoints hurting throughput)

Conclusion

Flink CDC is becoming the de facto standard for real-time data pipelines. It simplifies database-to-data-lake synchronization and provides high-quality data sources for real-time analytics, risk control, and recommendation. Continuous community investment (more databases, enhanced schema evolution, performance gains) will further cement its role in real-time data warehouse architectures.

Recommended learning path: Start with MySQL CDC → Integrate Kafka → Write to Iceberg/Hudi/Paimon → Build end-to-end real-time lakehouse pipeline .