CDC + Outbox Pattern: Production-Grade Cross-Database Sync with Spring Boot & Kafka

Why Traditional Sync Approaches Fail in Production

Cross-database synchronization becomes inevitable as microservices fragment into order, inventory, and risk databases that must exchange data within hundreds of milliseconds. Early approaches — dual-write in business code or scheduled table scans — break under load. Dual-write suffers from network timeouts, downstream restarts, and local transaction rollback timing mismatches, producing "local committed but downstream missed" or "downstream wrote first then local rolled back" anomalies. Adding Seata or XA preserves consistency but incurs heavy performance overhead and high migration cost for legacy systems. Async retry with exponential backoff creates retry queue backlogs during downstream outages; when the downstream recovers, the flood of retries collapses thread pools and connection pools in a classic cascade failure. Moreover, minute-level latency no longer meets business SLAs, while traditional async queues struggle to achieve sub-second end-to-end latency.

The core requirements are clear: don't touch the business transaction, guarantee local atomicity, capture changes in real time, and scale horizontally . CDC combined with the Outbox pattern is currently the engineering solution that best matches these demands.

Architecture Evolution: From Polling to CDC

Phase 1: Cron Polling

The earliest implementation used Cron jobs scanning update_time or cursor fields to fetch changes and push them to a message queue. High-frequency polling wastes CPU and I/O; changes within the time window are easily missed; multi-instance deployments require distributed locks to prevent duplicate processing. Operational cost is prohibitive and this approach is now obsolete.

Phase 2: Outbox Table

Introducing an outbox_event intermediate table lets business data and events be written in the same local transaction. A background process periodically polls unsent records and publishes them to the message queue. This solves the dual-write inconsistency root cause because local transaction ACID provides the guarantee. However, polling intervals leave latency stuck at several to tens of seconds, and cursor maintenance plus compensation logic must be built manually — not elegant enough.

Phase 3: CDC + Outbox Real-Time Capture

Debezium reads MySQL Binlog or PostgreSQL WAL directly. When the outbox_event table receives a new INSERT, Debezium's streaming parser uses Single Message Transforms (SMT) to convert the row into a Kafka Record immediately. The advantages are concrete:

Latency compressed to milliseconds : Binlog is append-only; Debezium streams reads. End-to-end latency typically stays within 100ms~300ms — orders of magnitude faster than polling.

Business completely unaware : The connector runs independently, zero intrusion into business code, no dependency on application-layer scheduled tasks.

Ordering guaranteed : Binlog write order strictly follows transaction commit order, naturally avoiding application-layer concurrency-induced reordering.

Core Pipeline Implementation: Wiring Spring Boot + Debezium + Kafka

3.1 Business Database Side: Writing Outbox Within the Transaction

The Outbox table schema need not be complex; it must carry routing information and the event payload. Production DDL:

CREATE TABLE outbox_event (id BIGINT AUTO_INCREMENT PRIMARY KEY, aggregate_id VARCHAR(64) NOT NULL, aggregate_type VARCHAR(32) NOT NULL, event_type VARCHAR(32) NOT NULL, payload JSON NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, INDEX idx_agg_type_id (aggregate_type, aggregate_id)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

The Spring Boot implementation has one critical rule: Outbox write must be bound to the same transaction as the business operation .

@Service@RequiredArgsConstructorpublic class OrderService {    private final JdbcTemplate jdbcTemplate;    @Transactional    public void createOrder(OrderCreateCmd cmd) {        // 1. Persist business table        jdbcTemplate.update(            "INSERT INTO t_order (id, user_id, amount, status) VALUES (?, ?, ?, ?)",            cmd.getOrderId(), cmd.getUserId(), cmd.getAmount(), "CREATED");        // 2. Persist Outbox table (same transaction context)        String payload = JsonUtils.toJson(cmd.toPayload());        jdbcTemplate.update(            "INSERT INTO outbox_event (aggregate_id, aggregate_type, event_type, payload) VALUES (?, ?, ?, ?)",            cmd.getOrderId(), "ORDER", "ORDER_CREATED", payload        );        // Binlog flushes only after transaction commit; Debezium reads it then    }}

Never move Outbox writes outside the transaction for "async" purposes, nor split them with @Async. If the transaction rolls back, the business row disappears but the Outbox record remains, and CDC will push dirty data downstream.

3.2 Debezium Connector Configuration

Production deployments should run Debezium on a dedicated Kafka Connect cluster. Key configuration with comments on common misconfigurations:

{  "name": "mysql-outbox-connector",  "config": {    "connector.class": "io.debezium.connector.mysql.MySqlConnector",    "database.hostname": "mysql-primary",    "database.port": "3306",    "database.user": "debezium",    "database.password": "xxxxxx",    "database.server.id": "184054",    "database.server.name": "db-server-1",    "database.history.kafka.bootstrap.servers": "kafka:9092",    "database.history.kafka.topic": "schema-changes.outbox",    "table.include.list": "shop.outbox_event",    "snapshot.mode": "initial",    "transforms": "outbox",    "transforms.outbox.type": "io.debezium.transforms.outbox.OutboxEventRouter",    "transforms.outbox.table.field.event.id": "id",    "transforms.outbox.table.field.event.key": "aggregate_id",    "transforms.outbox.table.field.event.type": "event_type",    "transforms.outbox.table.field.event.payload": "payload",    "transforms.outbox.table.fields.additional.placement": "aggregate_type:header:aggregate_type",    "value.converter": "org.apache.kafka.connect.json.JsonConverter",    "value.converter.schemas.enable": false  }}

Critical points: aggregate_id maps to Kafka key, ensuring all events for the same order land in the same partition — the foundation of ordering. payload maps directly to value; other metadata goes to header so downstream consumers don't parse wrapper JSON.

Verify transforms.outbox.type package path: Debezium 2.x moved it to io.debezium.transforms.outbox.OutboxEventRouter; older versions used io.debezium.transforms.OutboxEventRouter. Confirm against your dependency version before deployment.

3.3 Kafka Message Routing

After SMT transformation, the Kafka message looks like:

Topic: db-server-1.shop.outbox_eventKey: "ORD-20231024-0001"Headers: { "aggregate_type": "ORDER", "__debezium.outbox.event.type": "ORDER_CREATED" }Value: { "orderId": "ORD-20231024-0001", "userId": 1001, "amount": 299.00, "status": "CREATED" }

Downstream services should not hardcode topic names; route via the event_type header. If schema governance is strict, integrate Apicurio Registry + Avro and switch the connector's value.converter to AvroConverter, but this adds registry deployment and maintenance overhead — choose based on need.

Production Consistency Pitfalls That Must Be Guarded

Transaction Boundaries & Payload Size Control

The Outbox pattern's foundation is local transaction atomicity. The most common failure in sharded environments: if the business table and Outbox table reside on different physical shards, the local transaction cannot span them. Either co-locate the Outbox table with the business table on the same data source, or use middleware like ShardingSphere that supports cross-shard transactions.

Also, keep payload under 50KB. In MySQL Binlog ROW mode, large JSON inflates binlog volume, slows parsing, and increases primary I/O pressure. For large associated data, store only an ID in the payload and let downstream fetch details via callback.

Out-of-Order Handling & Partition Ordering

Kafka guarantees order only within a partition. Using aggregate_id as the key ensures events for the same aggregate root are ordered. Different orders are naturally parallel — this is desirable for throughput.

Rare cases demanding strict cross-entity ordering (e.g., global sequence numbers) would require forcing a single partition or an external sequencer, sacrificing scalability — an anti-pattern. The standard approach: include version or update_time in the payload; on consumption, discard events where version <= current without blocking the normal pipeline.

Idempotent Consumption & Dead Letter Queue (DLQ)

Without idempotency, a connector restart or network retransmission corrupts data. Database-level unique constraints are the hardest idempotency guarantee; don't rely solely on application code.

@KafkaListener(topics = "db-server-1.shop.outbox_event", groupId = "inventory-sync")public void handleOutbox(ConsumerRecord<String, String> record) {    String eventId = record.key();    String payload = record.value();    // Core: use DB unique index or ON DUPLICATE KEY for idempotency    // Production projects should encapsulate into an idempotency template to avoid scattering    int affected = jdbcTemplate.update(        "INSERT INTO t_inventory_sync (event_id, order_id, amount) VALUES (?, ?, ?) " +        "ON DUPLICATE KEY UPDATE amount = VALUES(amount)",        eventId, extractOrderId(payload), extractAmount(payload)    );    if (affected == 0) {        log.warn("Duplicate event consumed: {}", eventId);    }}

For exception handling, avoid custom retry loops — they easily deadlock. Use Spring Kafka's DefaultErrorHandler with RetryableTopic or route failed messages to a DLQ topic. The DLQ must carry the original offset and exception stack trace, with alerts routed to the on-call group. Production incidents abound where missing DLQ caused poison messages to stall consumer threads, backing up the entire topic.

Monitoring, Disaster Recovery & Daily Operations

How to Monitor Latency

CDC synchronization SLA lives or dies by monitoring. Debezium exposes JMX metrics; hook them to Micrometer and Prometheus. Focus on: debezium_mysql_metrics_milli_seconds_since_last_event: latest event latency; spike triggers immediate alert. kafka_consumer_lag: downstream consumption backlog; combined with binlog parsing latency, it pinpoints whether the connector or the consumer is the bottleneck.

Business-layer custom metric: Kafka消费时间戳 - outbox_event.created_at — the true business-perspective latency.

Grafana dashboards should overlay 99th-percentile latency, Consumer Lag curve, and Outbox table unprocessed row count (theoretically trending to zero). Issues become obvious at a glance.

Checkpoint Resume & Data Backfill

Connector sync state is persisted. Offsets live in offsets.storage.topic; schema changes in schema.history.topic. On node restart, streaming resumes automatically from the last committed Binlog Position — no manual intervention.

For historical backfill or downstream data corruption requiring replay, don't modify config and restart. Debezium 1.8+ introduced a Signaling Topic: send a JSON message to trigger an incremental snapshot safely and audibly:

# Send snapshot command to signaling topic (example)kafka-console-producer --topic debezium-signal --broker-list kafka:9092> {  "type": "execute-snapshot",  "data": {    "data-collections": ["shop.outbox_event"],    "type": "INCREMENTAL"  }}

During execution, reduce max.batch.size and increase poll.interval.ms to avoid saturating primary CPU. Narrow snapshot scope with snapshot.select.statement.overrides; full replays only after major promotions or architecture migrations.

Closing Notes

The CDC + Outbox combination essentially pushes the cross-database consistency problem from "application-layer coordination" down to "database log streaming." Spring Boot owns transaction atomicity; Debezium handles millisecond parsing and routing; Kafka provides the high-throughput event bus. Once running in production, you'll find it far lighter than wrestling distributed transactions and far more stable than polling scans.

When implementing, lock down three things:

Outbox write never leaves the business transaction — this is the baseline.

Downstream consumption must be idempotent; DB unique constraints beat code retries.

Latency monitoring and DLQ mechanism must be configured before go-live; don't wait for backlogs to appear before digging through logs.

Cloud vendors' managed DTS/DMS services are maturing, but the underlying thinking hasn't changed. Master this pattern and your data synchronization architecture stays sound regardless of component swaps. For specific connector tuning or Kafka consumer parameter questions, feel free to discuss — the production pits we've fallen into are largely covered by this paradigm.