The article examines the key challenges of real‑time data aggregation—data freshness, timely processing, and result visibility—and surveys common solutions such as timestamp‑based sync, CDC, full and incremental computation, storage formats, and trigger mechanisms.
The article reviews common cache‑side data‑sync patterns, highlights their inconsistency and data‑loss risks under high load, and presents JD's solution that combines Cache‑Aside, Change Data Capture, message queues, delayed consumption, versioning, and persistence to ensure eventual consistency between cache and relational databases.
This article explains how Flink SQL can be used to construct a unified stream‑batch ETL pipeline for data warehouses and data lakes, covering data integration, CDC support, streaming writes to Hive and Iceberg, and various join techniques such as regular, interval, and temporal joins.
The article details Apache Iceberg 0.11.0's core enhancements—including partition changes, SortOrder, extensive Flink and Spark integrations, CDC/Upsert support, hash‑based write distribution to reduce small files, and upcoming 0.12.0 roadmap—while providing practical SQL and API examples for data‑lake practitioners.
The article compares three approaches—synchronous dual write, asynchronous dual write with a message queue, and CDC via Canal—to keep Elasticsearch and a relational database consistent for cross‑database aggregated full‑text search, outlining their steps, advantages, and drawbacks.
This article describes how FluentSpeak replaced a DataX master‑slave pipeline with a CDC‑plus‑Delta Lake solution on Alibaba Cloud EMR, detailing architecture choices, streaming SQL merge logic, monitoring, challenges, and the resulting cost and latency improvements.
This article provides a step‑by‑step guide on configuring the MySQL CDC connector for Flink, covering Maven and SQL client dependencies, MySQL user setup, connector options, table creation via SQL and Stream API, key features, common issues, and practical troubleshooting tips.
This article explains Change Data Capture (CDC) and how Flink CDC can be used for incremental data synchronization, real‑time materialized views, audit logging, and CDC‑based joins, providing code examples, Maven dependencies, and SQL/Java snippets for MySQL and Kafka integrations.
This article explains how to replace daily offline MySQL‑to‑Hive sync with a CDC pipeline using Alibaba’s Canal to capture binlog events, Kafka for transport, and LinkedIn’s Camus (via a custom writer) to load data into Hive, detailing configuration and deployment steps.
Debezium is an open‑source distributed platform built on Apache Kafka that captures row‑level changes from databases via change data capture, providing source connectors, an optional embedded engine, and features like low‑latency streaming, snapshots, filtering, masking, and integration with various sink systems.
This article introduces four change‑data‑capture solutions—Canal, Maxwell, Databus, and Alibaba Cloud DTS—explaining their principles, processing steps, key features, and advantages for real‑time data replication and synchronization across diverse database systems in modern data engineering environments.
The article explains why achieving data consistency across multiple microservices is challenging, reviews the limitations of two‑phase commit, and presents practical techniques such as the Saga pattern, reconciliation, event logs, orchestration vs. choreography, and change‑data‑capture to keep distributed systems eventually consistent.
Facing MySQL performance bottlenecks at massive scale, Xiaomi built the LCSBinlog service that captures MySQL binlog events, streams them through the Talos platform, and writes to Kudu for real‑time BI, detailing architecture, job scheduling, consistency guarantees, use cases, and troubleshooting lessons.
Debezium is an open‑source distributed platform built on Apache Kafka that turns existing databases into real‑time event streams by capturing row‑level changes via change data capture, offering source and embedded connectors, flexible topic routing, and features such as snapshots, filtering, masking, and monitoring.
This article explains how CB‑SQL implements follower reads by using safe (closed) timestamps, describes the CT update mechanism with a Minimum Proposal Tracker, and discusses routing, replica read validation, timestamp forwarding, range split/merge handling, and recovery strategies for consistent distributed reads.
This article explores the core components of data systems, the evolution toward lightweight, intelligent big data architectures, the distinction between primary and secondary storage, challenges of data replication, and how Alibaba Cloud's Tablestore implements advanced features such as storage‑compute separation, CDC, and multi‑model indexing for scalable, cost‑effective structured big data storage.
The article explains CB‑SQL’s distributed changefeed architecture, its CDC implementation, the challenges of horizontal scalability and transactional ordering, and the innovative RangeFeed mechanism that enables ordered row‑level streams, resolved timestamps, and seamless integration with external systems like Kafka.
The article introduces CB‑SQL, JD Retail’s next‑generation elastic database based on CockroachDB, and comprehensively explains its overview, layered architecture, storage replication, cluster management, MVCC, hybrid logical clocks, lock‑free distributed transactions, CDC changefeeds, and SQL interface support.
The article presents DTLE, an open‑source MySQL data‑transfer middleware that extends replication capabilities with high‑performance CDC, multi‑topology support, cloud‑to‑cloud synchronization, and robust cluster management, while comparing it with other open‑source solutions and showcasing real‑world demos.
The article presents a CDC solution that combines Alibaba’s Canal for real‑time MySQL binlog capture into Kafka with LinkedIn’s Camus for hourly Kafka‑to‑Hive loading, then merges snapshots and incremental binlog data to accurately and efficiently rebuild ODS tables, supporting sharding and delete events.
This article details the challenges of moving from SQL Server to MySQL, compares offline and online migration approaches, presents a complete toolbox—including ETL utilities, consistency checkers, CDC configuration, and rollback mechanisms—and shares practical lessons from a large‑scale production migration.
This article explains how to achieve near‑zero‑downtime online migration from SQL Server to MySQL by leveraging Change Data Capture (CDC) and the Alibaba Yugong tool, covering CDC principles, configuration, performance tuning, and practical migration steps.
This article details how Vimur, a Kafka‑based real‑time data pipeline, addresses the challenges of service splitting, database sharding, data migration, and synchronization by using CDC, a unified Avro format, and a change distribution platform to support search indexing, cache refresh, and reactive architectures.
This article explains the design and implementation of the DWS platform, detailing its log‑driven architecture with Dbus, Wormhole, and Swifts, the technical choices behind real‑time data extraction, transformation, and delivery, and real‑world use cases in finance.
This article explains why DB2 customers must upgrade to avoid EOS risks, compares replication options, and provides a detailed CDC‑based step‑by‑step process to achieve minute‑level downtime, zero performance impact, and fast rollback for large‑scale DB2 version migrations.
