Resolving Flink CDC server_id Collisions: Production Debugging Walkthrough

Background

A data engineering team configured Flink CDC tasks to sync MySQL data to StarRocks in real time. To accelerate setup, they copied existing task code and modified only the database/table settings, but neglected to change the server_id. After launch, monitoring alerts showed continuously growing business latency. Investigation revealed a server_id conflict causing task anomalies.

Core Reason: Flink CDC as a Logical MySQL Replica

Flink CDC acts as a MySQL logical replica — it uses Debezium to simulate the MySQL replica protocol and request binlog synchronization. The MySQL primary identifies replicas solely by two identifiers: server_id — an integer that must be unique across all replicas (physical + logical) under the same primary. server_uuid — a string generated by Debezium based on server.name; normally unique, but cloning tasks can cause duplication.

If any of the following occur, the primary refuses binlog sync:

Two Flink CDC tasks share the same server_id.

A physical replica's server_id or server_uuid collides with the CDC task.

On restart, a CDC task fails to regenerate a new server_uuid (rare, version-dependent).

Production-Grade Troubleshooting Steps

Step 1: Locate the Conflict Source

(1) Inspect All Replica Connections on the Primary

Log into the MySQL primary and run:

-- View all binlog dump connections (Command = Binlog Dump indicates a replica)
SHOW PROCESSLIST;

(2) Collect Conflicting Replica Identifiers

Physical replica (e.g., 192.168.1.101): log in and execute:

SHOW VARIABLES LIKE 'server_id';  -- e.g., returns 101
SHOW VARIABLES LIKE 'server_uuid'; -- e.g., returns uuid-101

Other Flink CDC task (e.g., 192.168.1.102): check its configuration for debezium.database.server.id (e.g., 200) and debezium.database.server.name (e.g., cdc-task-01, which determines server_uuid).

(3) Examine the Failing CDC Task Configuration

Confirm the erroneous server_id and server_name:

-- Example: current task's conflicting config
CREATE TABLE error_cdc_source (
  ...
) WITH (
  ...
  'debezium.database.server.id' = '200',  -- duplicates another CDC task
  'debezium.database.server.name' = 'cdc-task-01'  -- duplicate
);

Step 2: Assign Unique Identifiers

(1) Modify server_id (Mandatory)

Rule: server_id must be an integer 1–4294967295, distinct from the primary, all physical replicas, and other CDC tasks.

Recommended planning by business module + sequence:

Physical replicas: 100–199

User-module CDC: 200–299

Order-module CDC: 300–399

Example: if the task handles orders and 301 is free:

'debezium.database.server.id' = '301'  -- unique and unoccupied

(2) Modify server_name (Prevents server_uuid Collision)

server_uuid

is auto-generated by Debezium using server_name (default format: server_name + random_string). Duplicate server_name can lead to duplicate server_uuid.

Naming convention: database-table-taskseq, e.g.:

'debezium.database.server.name' = 'order-db-orders-cdc-02'  -- unique across tasks

(3) Clear Old State (Critical)

Flink CDC stores historical server_id and offset in checkpoint state. Without clearing, a restart will reload the old config and the conflict persists.

Method 1: Delete the task's state directory in the Flink state backend (e.g., hdfs:///flink/checkpoints/{job-id}/).

Method 2: Recreate the table in Flink SQL with a new name, or regenerate the JobID in code to avoid reusing old state.

Step 3: Verify Resolution

Flink logs: No same server_uuid/server_id errors; CDC source operator shows RUNNING.

MySQL primary connections: Run SHOW PROCESSLIST; again — the CDC task appears with the Flink node IP as Host, Command = Binlog Dump, and no duplicate server_id entries.

Data sync: Downstream sinks (Kafka, StarRocks) receive data normally, no gaps or duplicate replication.

Prevention: Establish a server_id Management Ledger

Maintain a registry recording every replica's identifiers (physical + logical):

Before adding a new CDC task, consult the ledger to pick an unused server_id, eliminating conflicts at the source.

Summary

The fix hinges on uniqueness : investigate to find the duplicate server_id or server_uuid, assign a unique identifier to the CDC task, and purge old checkpoint state so the new config takes effect. Coupled with a disciplined server_id registry, this class of issue is permanently avoided.