BestHub
Sign in
Tagged articles

CDC

145 articles · Page 1 of 2
ITPUB
ITPUB
Aug 17, 2026 · Databases

How a Veteran DBA Tackles Financial Multi‑Database and Big Data Architecture

In this interview, senior DBA Yao Wei shares over a decade of hands‑on experience designing, testing, and operating heterogeneous database ecosystems for retail, internet, and financial services, detailing pre‑emptive DBA involvement, compatibility pitfalls, high‑availability strategies, automated validation pipelines, and migration to domestic databases.

AutomationCDCData Migration
0 likes · 32 min read
How a Veteran DBA Tackles Financial Multi‑Database and Big Data Architecture
Yumin Fish Harvest
Yumin Fish Harvest
Aug 13, 2026 · Databases

How to Choose Storage for Query Services: Relational DB, Search, OLAP, and Time‑Series Models

The article walks through a systematic method for selecting the right storage technology—relational OLTP, search engine, OLAP, or time‑series—based on query intent, data volume, latency tolerance, and consistency needs, using an e‑commerce “optimal store” scenario to illustrate the decision process and practical trade‑offs.

CDColapread-model
0 likes · 30 min read
How to Choose Storage for Query Services: Relational DB, Search, OLAP, and Time‑Series Models
Data Integration and Governance
Data Integration and Governance
Jul 31, 2026 · Operations

Ensuring Data Sync Is No‑Duplicate, No‑Loss, Accurate – Full‑Load, Incremental, Validation, Exception Handling

The article explains why reliable data synchronization must answer three questions—preventing duplicates, avoiding missing data, and preserving correct content—and details practical solutions for full‑load and incremental sync, idempotent writes, CDC, handling late, out‑of‑order and breakpoint failures, multi‑layer validation, and categorized exception recovery.

CDCData synchronizationException Handling
0 likes · 15 min read
Ensuring Data Sync Is No‑Duplicate, No‑Loss, Accurate – Full‑Load, Incremental, Validation, Exception Handling
LuTiao Programming
LuTiao Programming
Jul 25, 2026 · Backend Development

Why Delayed Double Delete Fails: A Hierarchical Java Design for MySQL‑Redis Consistency

The article analyzes why MySQL and Redis cannot guarantee strong consistency with simple cache‑aside patterns, explains the pitfalls of delayed double delete, and presents a tiered Java design—including transaction‑after‑commit deletion, retryable invalidation, CDC/Outbox pipelines, TTL safeguards, and multi‑level cache considerations—to achieve reliable cache consistency.

CDCCache invalidationMySQL
0 likes · 22 min read
Why Delayed Double Delete Fails: A Hierarchical Java Design for MySQL‑Redis Consistency
System Architect Go
System Architect Go
Jul 23, 2026 · Databases

Building a PostgreSQL WAL‑Based CDC System: Principles and Engineering Implementation

This article explains how to capture and stream row‑level changes from PostgreSQL using its Write‑Ahead Log, compares the approach with MySQL binlog CDC, and provides a step‑by‑step guide—including configuration, logical decoding, snapshot handling, replication slots, and production‑grade correctness considerations—to build a reliable CDC pipeline.

CDCData PipelineLogical Decoding
0 likes · 29 min read
Building a PostgreSQL WAL‑Based CDC System: Principles and Engineering Implementation
Cloud Architecture
Cloud Architecture
Jul 16, 2026 · Backend Development

Beyond Delayed Double Delete: How CDC Closed‑Loop Governance Solves Cache Consistency

The article dissects why the classic "update‑DB‑then‑delete‑cache" or its reverse is only a probability fix for cache inconsistency, demonstrates the failure modes of delayed double delete under high load and replication lag, and presents a production‑grade solution built on Binlog CDC with versioning, four‑plane governance, and robust error handling to achieve reliable cache synchronization.

CDCVersioningbackend
0 likes · 36 min read
Beyond Delayed Double Delete: How CDC Closed‑Loop Governance Solves Cache Consistency
Cloud Architecture
Cloud Architecture
Jun 30, 2026 · Backend Development

RocketMQ Transactional Messaging in Practice: From Half Message to Production‑Grade Consistency

The article analyses why sending a message is easy but guaranteeing end‑to‑end consistency across databases, brokers, networks and services is hard, explains the exact problems RocketMQ transactional messages solve, compares them with Outbox and CDC, and provides a complete production‑grade design, implementation, monitoring and tuning guide.

CDCIdempotencyRocketMQ
0 likes · 44 min read
RocketMQ Transactional Messaging in Practice: From Half Message to Production‑Grade Consistency
Niu Liu
Niu Liu
Jun 29, 2026 · Big Data

Designing a Real‑Time Data Warehouse Backend with Flink and Paimon for SMBs

The article walks through the architecture and core workflow of a lightweight real‑time data warehouse built on Flink and Paimon, covering service‑layer refactoring, CDC task lifecycle, dual Flink submission modes, metadata synchronization, and layered security measures for small‑to‑medium enterprises.

Backend ArchitectureCDCFlink
0 likes · 11 min read
Designing a Real‑Time Data Warehouse Backend with Flink and Paimon for SMBs
Cloud Architecture
Cloud Architecture
Jun 26, 2026 · Databases

PostgreSQL Backup & Replication Guide: WAL Basics to Production‑Ready HA, DR, and Recovery

This comprehensive guide walks through PostgreSQL's backup and replication architecture, explaining WAL fundamentals, the distinction between backup, high‑availability, disaster‑recovery and CDC, and provides step‑by‑step configurations, performance trade‑offs, monitoring queries, Kubernetes deployment tips, and practical recovery‑drill procedures for production environments.

BackupCDCHA
0 likes · 40 min read
PostgreSQL Backup & Replication Guide: WAL Basics to Production‑Ready HA, DR, and Recovery
Cloud Architecture
Cloud Architecture
Jun 22, 2026 · Databases

Opening the Hood of PostgreSQL WAL: From Transaction Logs to Production‑Grade HA, Replication, and Performance Tuning

This article dissects PostgreSQL's Write‑Ahead Logging (WAL), explaining how it underpins transaction durability, replication, CDC, point‑in‑time recovery, and commit latency, and provides a step‑by‑step guide to architecture design, parameter tuning, troubleshooting, containerization, backup strategies, and operational checklists for production‑grade deployments.

BackupCDCHigh Availability
0 likes · 37 min read
Opening the Hood of PostgreSQL WAL: From Transaction Logs to Production‑Grade HA, Replication, and Performance Tuning
Past Memory Big Data
Past Memory Big Data
Jun 22, 2026 · Big Data

What’s New in Apache Spark 4.2? Core Features and Architecture Evolution

Apache Spark 4.2 introduces a lightweight Spark Connect architecture, native AI integration, enhanced Metrics View for unified semantics, Arrow‑first performance gains, advanced SQL extensions like vector search and QUALIFY, robust geospatial support, and a revamped streaming engine with auto CDC and sub‑millisecond state cleanup.

Apache SparkArrowCDC
0 likes · 13 min read
What’s New in Apache Spark 4.2? Core Features and Architecture Evolution
AI Engineer Programming
AI Engineer Programming
Jun 21, 2026 · Artificial Intelligence

RAG Data Governance: Incremental Sync and Consistency (Part 1)

The article explains how additions, updates, and deletions affect a vector store differently, outlines three layers of incremental synchronization—change detection, change handling, and service stability—and compares timestamp polling, content‑hash diffing, and CDC while discussing consistency models and conflict resolution in distributed vector databases.

CDCData GovernanceRAG
0 likes · 16 min read
RAG Data Governance: Incremental Sync and Consistency (Part 1)
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 18, 2026 · Big Data

How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification

In the AI era, enterprises need a data foundation that supports both low‑latency streaming and long‑term analytics, and the combination of Kafka, Iceberg and object storage is emerging as a preferred solution; by moving ingestion capabilities closer to the message layer and eliminating external ETL jobs, a "zero‑ETL" approach reduces architectural complexity, improves consistency, and streamlines schema evolution and small‑file management.

CDCData LakeIceberg
0 likes · 27 min read
How AI-Driven Real-Time Data Lakes Are Ditching ETL: A Kafka‑to‑Iceberg Architecture Simplification
Cloud Architecture
Cloud Architecture
May 13, 2026 · Backend Development

Mastering Spring Boot Data Access: From ORM and Caching to Search and Distributed Consistency

This extensive guide redesigns Spring Boot data‑access for high‑traffic e‑commerce, explaining why traditional JPA‑Redis‑Elasticsearch thinking fails, then detailing a multimodal architecture that assigns strong‑consistency, hot‑read, document, and search responsibilities to MySQL, Redis, MongoDB and Elasticsearch, with production‑grade code, CDC pipelines, distributed‑transaction patterns, caching strategies, observability, and cloud‑native deployment.

CDCElasticsearchJPA
0 likes · 49 min read
Mastering Spring Boot Data Access: From ORM and Caching to Search and Distributed Consistency
Data Integration and Governance
Data Integration and Governance
May 7, 2026 · Big Data

Still Using Traditional Data Warehouses? A Complete Guide to Real‑Time Data Warehousing

Traditional batch‑oriented data warehouses can’t keep up with AI‑driven, second‑level business needs, so the article explains what a real‑time data warehouse is, its key technical traits, business benefits such as faster decision making and cost savings, and provides a step‑by‑step implementation roadmap.

CDCData GovernanceData Integration
0 likes · 15 min read
Still Using Traditional Data Warehouses? A Complete Guide to Real‑Time Data Warehousing
Cloud Architecture
Cloud Architecture
Apr 7, 2026 · Databases

Practical Guide to Safely Adding a Column to a Ten‑Million‑Row Order Table: Principles, Architecture, and Production Rollout

This article presents a production‑grade solution for adding a new promotion_type column to a tens‑of‑millions‑row order table, covering MySQL DDL risks, multiple migration strategies, a step‑by‑step implementation using a new table, CDC‑based incremental sync, back‑fill, verification, gray rollout, and rollback procedures.

BackfillCDCGray Deployment
0 likes · 38 min read
Practical Guide to Safely Adding a Column to a Ten‑Million‑Row Order Table: Principles, Architecture, and Production Rollout
Cloud Architecture
Cloud Architecture
Mar 29, 2026 · Backend Development

Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing

This article demystifies Kafka's ordering guarantees, explains the difference between partition‑level, key‑level, and global ordering, and provides a complete, production‑grade guide—including architecture, configuration, code samples, and best‑practice patterns—to achieve reliable ordered processing while maintaining high throughput and scalability.

CDCKafkaOrdering
0 likes · 35 min read
Kafka Ordered Writes and Reads: The Full Truth Behind Message Sequencing
TechVision Expert Circle
TechVision Expert Circle
Mar 17, 2026 · Industry Insights

How Leading CTOs Build an End-to-End Data Pipeline from Collection to AI‑Driven Insight

The article outlines a full‑stack data architecture for 2026, covering edge‑preprocessed collection, unified observability with OpenTelemetry, CDC‑based real‑time sync, a Kafka‑Flink‑Iceberg processing stack, Data Mesh governance, and AI‑Agent powered insight, while offering practical adoption paths and pitfalls to avoid.

AI AgentCDCData Mesh
0 likes · 14 min read
How Leading CTOs Build an End-to-End Data Pipeline from Collection to AI‑Driven Insight
ITPUB
ITPUB
Feb 13, 2026 · Big Data

Real‑Time Sync of New MySQL Tables to Doris Using Flink CDC

This article explains how to extend a Flink CDC job that already syncs an entire MySQL database to Doris so that newly created tables are automatically created in Doris in real time, using the CdcTools utility, side‑output streams, and asynchronous I/O.

CDCCdcToolsDoris
0 likes · 9 min read
Real‑Time Sync of New MySQL Tables to Doris Using Flink CDC
ITPUB
ITPUB
Jan 22, 2026 · Backend Development

Sync New MySQL Tables to Doris in Real‑Time with Flink CDC and CdcTools

This article explains how to use Flink CDC together with the CdcTools utility to automatically capture newly created MySQL tables and synchronize both their schema and data to a Doris database in real time, covering the required code, side‑output handling, async execution, and a special delete‑sign field.

Async IOCDCDoris
0 likes · 10 min read
Sync New MySQL Tables to Doris in Real‑Time with Flink CDC and CdcTools
ITPUB
ITPUB
Jan 18, 2026 · Databases

From Full Sync to Real‑Time CDC: Building Scalable Order Data Pipelines

An e‑commerce junior developer tackles the challenge of regularly syncing order data to a data warehouse, evolving from naïve full‑table copies to incremental sync, batch processing, cursor‑based pagination, performance tuning, and finally a real‑time CDC‑plus‑message‑queue architecture, while addressing reliability, ordering, and scaling issues.

CDCCursorbatch
0 likes · 13 min read
From Full Sync to Real‑Time CDC: Building Scalable Order Data Pipelines
Ctrip Technology
Ctrip Technology
Nov 20, 2025 · Big Data

How Ctrip Achieved Minute‑Level Real‑Time Analytics with Flink CDC & Apache Paimon

Ctrip transformed its traditional T+1 offline warehouse into a near‑real‑time lakehouse by integrating Flink CDC with Apache Paimon, designing a two‑stage CDC ingestion, optimizing performance, implementing dynamic updates, and deploying the solution across multiple business scenarios, achieving minute‑level latency, reduced costs, and faster data‑driven decisions.

CDCData EngineeringFlink
0 likes · 27 min read
How Ctrip Achieved Minute‑Level Real‑Time Analytics with Flink CDC & Apache Paimon
dbaplus Community
dbaplus Community
Nov 12, 2025 · Databases

Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce

This guide walks a new e‑commerce developer through the evolution of order data synchronization—from naïve full‑table loads, through incremental and batch strategies, cursor‑based pagination, performance tuning, and finally to real‑time CDC with message queues—highlighting pitfalls and practical solutions.

Batch ProcessingCDCCursor Pagination
0 likes · 12 min read
Mastering Data Sync: From Full Loads to Real‑Time CDC in E‑Commerce
Smart Sea Tide
Smart Sea Tide
Nov 4, 2025 · Big Data

Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink

The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.

Apache IcebergCDCData Lake
0 likes · 18 min read
Implementing an Integrated Data Lake and Lakehouse Architecture with Apache Iceberg and Flink
StarRocks
StarRocks
Jul 1, 2025 · Big Data

How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics

Suixingfu rebuilt its payment data pipeline by replacing a fragmented Lambda stack with a unified Porter CDC + StarRocks + Elasticsearch architecture, achieving three‑fold query speed, ten‑fold analytics efficiency, 20% storage reduction, and sub‑second data‑capture latency across high‑concurrency, ad‑hoc, and batch workloads.

CDCData WarehouseFlink
0 likes · 14 min read
How StarRocks Boosted Suixingfu’s Real‑Time Data Platform: 3× Faster Queries & 10× Faster Analytics
Big Data Technology Tribe
Big Data Technology Tribe
Jun 22, 2025 · Cloud Native

How to Ensure Consistent State in Event‑Driven Microservices: 3 Proven Patterns

This article explains the challenges of maintaining data consistency in distributed, event‑driven microservice architectures and introduces three practical patterns—Outbox, Original Event Handling, and Self‑Read—to guarantee reliable state synchronization across services, even when failures occur.

CDCdistributed transactionsevent-driven-architecture
0 likes · 6 min read
How to Ensure Consistent State in Event‑Driven Microservices: 3 Proven Patterns
DataFunSummit
DataFunSummit
Apr 1, 2025 · Big Data

Understanding Flink CDC 3.3: Features, Improvements, and Future Plans

This article provides a comprehensive overview of Flink CDC 3.3, detailing its CDC fundamentals, new connectors, Transform module enhancements, asynchronous snapshot splitting, community adoption, and upcoming roadmap for broader ecosystem support and batch‑mode execution.

Big DataCDCChange Data Capture
0 likes · 15 min read
Understanding Flink CDC 3.3: Features, Improvements, and Future Plans
Big Data Technology Architecture
Big Data Technology Architecture
Mar 1, 2025 · Big Data

Core Principles and Practical Guide to Flink CDC

This article explains CDC fundamentals, details Flink CDC's architecture and advantages, provides setup steps, code examples for SQL and DataStream APIs, discusses performance tuning, consistency, common issues, and typical real‑time data integration scenarios.

CDCChange Data CaptureDebezium
0 likes · 7 min read
Core Principles and Practical Guide to Flink CDC
DataFunSummit
DataFunSummit
Feb 24, 2025 · Big Data

Building Real-Time Data Synchronization Pipelines with Apache SeaTunnel

Apache SeaTunnel is an open‑source, distributed data integration platform that enables efficient real‑time data synchronization across diverse sources and destinations, supporting both streaming and batch processing, with detailed architecture, connector plugins, CDC handling, transform capabilities, and deployment strategies for large‑scale data pipelines.

Apache SeaTunnelCDCData pipelines
0 likes · 34 min read
Building Real-Time Data Synchronization Pipelines with Apache SeaTunnel
macrozheng
macrozheng
Feb 24, 2025 · Databases

Mastering MySQL to Elasticsearch Sync: 4 Strategies & Top Migration Tools

This article explores four practical methods for synchronizing MySQL data to Elasticsearch—including synchronous and asynchronous double writes, SQL extraction, and binlog real‑time replication—while reviewing popular migration tools such as Canal, Alibaba DTS, and Databus to help you choose the right solution.

CDCCanalDTS
0 likes · 13 min read
Mastering MySQL to Elasticsearch Sync: 4 Strategies & Top Migration Tools
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jan 27, 2025 · Big Data

Unlock Real-Time Data Sync with Flink CDC: YAML Integration, Transform & Route Explained

This article summarizes an advanced Flink CDC presentation, covering Flink CDC fundamentals, real‑time Flink integration, CDC‑YAML core capabilities, supported sync links, Transform and Route modules, monitoring metrics, schema‑change strategies, typical use cases, performance optimizations, demo implementations, and future development plans.

CDCData IntegrationFlink
0 likes · 20 min read
Unlock Real-Time Data Sync with Flink CDC: YAML Integration, Transform & Route Explained
Tencent Advertising Technology
Tencent Advertising Technology
Dec 6, 2024 · Big Data

Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent

Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.

Big DataCDCData Lake
0 likes · 25 min read
Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent
Su San Talks Tech
Su San Talks Tech
Jul 26, 2024 · Databases

Mastering MySQL‑to‑Elasticsearch Sync: 4 Strategies & Top Migration Tools

This guide compares four MySQL‑to‑Elasticsearch synchronization methods—synchronous dual‑write, asynchronous MQ‑based dual‑write, timer‑driven SQL extraction, and real‑time Binlog replication—and reviews popular CDC tools such as Canal, Alibaba Cloud DTS, Databus, and others to help you choose the right solution.

CDCCanalDTS
0 likes · 13 min read
Mastering MySQL‑to‑Elasticsearch Sync: 4 Strategies & Top Migration Tools
IT Services Circle
IT Services Circle
Jun 12, 2024 · Databases

MySQL to Elasticsearch Data Synchronization: Strategies and Tool Selection

This article reviews four common MySQL‑to‑Elasticsearch synchronization methods—synchronous dual‑write, asynchronous dual‑write via MQ, timer‑based SQL extraction, and real‑time Binlog replication—evaluates their pros and cons, and compares popular migration tools such as Canal, Alibaba DTS, Databus and others.

CDCData Migration ToolsData synchronization
0 likes · 11 min read
MySQL to Elasticsearch Data Synchronization: Strategies and Tool Selection
Su San Talks Tech
Su San Talks Tech
Jun 10, 2024 · Databases

Mastering MySQL‑to‑Elasticsearch Sync: 4 Strategies & Top Migration Tools

This article compares four MySQL‑to‑Elasticsearch synchronization methods—synchronous dual‑write, asynchronous dual‑write, SQL extraction, and Binlog‑based real‑time sync—evaluates their pros and cons, and reviews popular migration tools such as Canal, Alibaba DTS, Databus, Flink, CloudCanal, Maxwell, and DRDS.

CDCData synchronizationElasticsearch
0 likes · 14 min read
Mastering MySQL‑to‑Elasticsearch Sync: 4 Strategies & Top Migration Tools
DataFunTalk
DataFunTalk
May 16, 2024 · Big Data

Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon

This article presents UCloud's USDP‑based streaming data lake warehouse solution that leverages Flink for real‑time processing and Paimon for lake storage, detailing its architecture, advantages, practical scenarios, and providing complete SQL and Flink CDC code snippets for end‑to‑end implementation.

CDCData LakeFlink
0 likes · 27 min read
Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon
DataFunSummit
DataFunSummit
Mar 25, 2024 · Big Data

Exploring Real-Time Data Lake Practices at Kangaroo Cloud

This article shares Kangaroo Cloud's exploration and practice of a real-time data lake, covering background, data lake concepts, challenges, solution architecture using the Shuzhan platform with Iceberg/Hudi, CDC ingestion, small file handling, cross-cluster ingestion, materialized view acceleration, and future development plans.

CDCCross-Cluster IngestionHudi
0 likes · 12 min read
Exploring Real-Time Data Lake Practices at Kangaroo Cloud
DataFunSummit
DataFunSummit
Feb 20, 2024 · Big Data

BitSail Open‑Source Data Integration Engine: Architecture, New Features, CDC Solutions and Future Outlook

This article introduces ByteDance's open‑source data integration engine BitSail, covering its background, layered architecture, recent feature enhancements, automated testing framework, CDC‑based full‑library synchronization solutions, and future development plans for connectors and real‑time data consistency.

Big DataCDCData Integration
0 likes · 12 min read
BitSail Open‑Source Data Integration Engine: Architecture, New Features, CDC Solutions and Future Outlook
ITPUB
ITPUB
Dec 24, 2023 · Backend Development

Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines

This article explains how Kafka serves as a high‑throughput, durable messaging system, a reliable storage layer, a log‑aggregation hub, a stream‑processing engine, and a core component for CDC, system migration, monitoring, and event‑sourcing architectures.

CDCKafkaStreaming
0 likes · 9 min read
Why Kafka Is the Backbone of Modern Messaging, Streaming, and Data Pipelines
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 28, 2023 · Big Data

Apache Paimon for CDC: Low‑Cost, Low‑Latency Data Lake Ingestion and Performance Comparison with Hive and Hudi

This article explains how Apache Paimon simplifies CDC data lake ingestion with one‑click, low‑cost, low‑latency pipelines, details its architecture and tag‑based Hive compatibility, provides best‑practice configurations, and presents benchmark results showing Paimon outperforming Hive and Hudi in both write and query performance.

Apache PaimonCDCData Lake
0 likes · 14 min read
Apache Paimon for CDC: Low‑Cost, Low‑Latency Data Lake Ingestion and Performance Comparison with Hive and Hudi
Alibaba Cloud Native
Alibaba Cloud Native
Nov 23, 2023 · Cloud Native

How CDC + Serverless Functions Enable Real‑Time ETL in Cloud Native Architectures

This article explains how Alibaba Cloud's Serverless Function Compute combined with Database Change Data Capture (CDC) creates a complete, real‑time ETL pipeline, detailing the ETL model, DTS integration, architecture components, event‑driven processing, and practical use cases such as OLTP‑to‑OLAP data flow.

Alibaba CloudCDCData Integration
0 likes · 10 min read
How CDC + Serverless Functions Enable Real‑Time ETL in Cloud Native Architectures
Java High-Performance Architecture
Java High-Performance Architecture
Sep 28, 2023 · Databases

How to Use Debezium for MySQL CDC in Spring Boot Without Adding Extra Middleware

Learn how to capture MySQL data changes using Debezium's CDC capabilities within a Spring Boot application, avoiding heavyweight message brokers by leveraging binlog monitoring, configuring connectors, handling snapshots, and processing change events for use cases like cache invalidation, data integration, and simplifying monolithic architectures.

CDCData IntegrationDebezium
0 likes · 24 min read
How to Use Debezium for MySQL CDC in Spring Boot Without Adding Extra Middleware
dbaplus Community
dbaplus Community
Sep 24, 2023 · Backend Development

How to Sync MySQL Binlog to Elasticsearch Using Canal and RocketMQ

Learn step‑by‑step how to configure Alibaba’s open‑source Canal to capture MySQL binlog changes, route them through RocketMQ, and index the data into Elasticsearch, covering cluster mode, MySQL and Elasticsearch setup, Canal properties, and consumer implementation details.

CDCCanalMySQL
0 likes · 9 min read
How to Sync MySQL Binlog to Elasticsearch Using Canal and RocketMQ
Java Interview Crash Guide
Java Interview Crash Guide
Aug 14, 2023 · Big Data

Unlocking Change Data Capture with Debezium in Spring Boot – No Extra Middleware Needed

This article explains how small web projects can avoid heavyweight message middleware by using CDC technology, specifically Debezium, to monitor MySQL binlog changes, outlines why Debezium outperforms alternatives like Canal, and provides step‑by‑step Spring Boot integration with configuration, code samples, and practical use‑case scenarios.

CDCChange Data CaptureDebezium
0 likes · 22 min read
Unlocking Change Data Capture with Debezium in Spring Boot – No Extra Middleware Needed
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 4, 2023 · Big Data

Building a Real‑Time Streaming Data Warehouse with Paimon on Kubernetes for Supply‑Chain Logistics

This article presents a step‑by‑step guide on how the logistics provider Haicheng Bangda implemented a streaming data warehouse using Paimon, Flink CDC, and Kubernetes, covering business background, architecture choices, environment setup, SQL examples, troubleshooting tips, and future roadmap for their digital transformation.

Big DataCDCData Warehouse
0 likes · 27 min read
Building a Real‑Time Streaming Data Warehouse with Paimon on Kubernetes for Supply‑Chain Logistics
DataFunSummit
DataFunSummit
May 28, 2023 · Big Data

Apache Hudi: Capabilities, Architecture, Use Cases, and Future Outlook

This article introduces Apache Hudi as a next‑generation streaming data‑lake platform, explains its core concepts, architecture, and table types, and showcases real‑world use cases at Tencent such as CDC ingestion, minute‑level real‑time warehousing, streaming analytics, multi‑stream joins, ad attribution, and stream‑to‑batch processing, while also outlining future directions.

Apache HudiCDCData Lake
0 likes · 16 min read
Apache Hudi: Capabilities, Architecture, Use Cases, and Future Outlook
ITPUB
ITPUB
Apr 26, 2023 · Databases

Mastering Change Data Capture: Open‑Source Tools and How to Choose the Right One

This article explains the concept of Change Data Capture (CDC), outlines its common use cases, compares the main technical approaches—including timestamps, data diff, triggers, and log‑based methods—and reviews popular open‑source CDC solutions and their database‑specific configuration requirements.

CDCChange Data CaptureData Integration
0 likes · 15 min read
Mastering Change Data Capture: Open‑Source Tools and How to Choose the Right One
TAL Education Technology
TAL Education Technology
Feb 16, 2023 · Big Data

Step‑by‑Step Guide to Syncing Canal Data to Elasticsearch

This article provides a comprehensive, hands‑on tutorial for configuring Alibaba Canal and its client‑adapter to capture MySQL binlog changes and synchronize them into Elasticsearch, covering environment setup, Docker commands, YAML configuration files, index mapping, adapter startup, and common troubleshooting scenarios.

CDCCanalConfiguration
0 likes · 26 min read
Step‑by‑Step Guide to Syncing Canal Data to Elasticsearch
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 28, 2022 · Big Data

Flink 1.16 Highlights: Adaptive Batch Scheduling, Speculative Execution, Hybrid Shuffle, Dynamic Partition Pruning, Hive SQL Migration, Checkpoint Enhancements, CDC Integration, and Table Store

Flink 1.16 introduces adaptive batch scheduling, speculative execution, hybrid shuffle, dynamic partition pruning, improved Hive SQL compatibility, advanced checkpoint mechanisms including changelog backend, and integrates CDC with Kafka and Table Store, offering faster, more stable, and easier-to-use stream‑batch processing capabilities.

Big DataCDCCheckpoint
0 likes · 8 min read
Flink 1.16 Highlights: Adaptive Batch Scheduling, Speculative Execution, Hybrid Shuffle, Dynamic Partition Pruning, Hive SQL Migration, Checkpoint Enhancements, CDC Integration, and Table Store
ITPUB
ITPUB
Dec 18, 2022 · Big Data

How to Build a Real‑Time Data Warehouse with EasyData: A Step‑by‑Step Guide

Learn how to design and implement a real‑time data warehouse for an app’s AB‑test monitoring using EasyData, covering data flow layers, CDC task creation, stream table registration, Flink SQL processing, and BI reporting, with detailed steps, code snippets, and practical tips.

CDCEasyDataFlink
0 likes · 13 min read
How to Build a Real‑Time Data Warehouse with EasyData: A Step‑by‑Step Guide
DataFunSummit
DataFunSummit
Dec 2, 2022 · Big Data

BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities

BitSail, ByteDance’s open‑source data integration engine, unifies batch, streaming, and incremental data synchronization across heterogeneous sources, detailing its evolution from early Flink‑based prototypes to a mature, plugin‑driven architecture with multi‑engine support, low‑cost co‑development, and robust CDC lakehouse capabilities.

Big DataCDCFlink
0 likes · 19 min read
BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities
DataFunTalk
DataFunTalk
Nov 6, 2022 · Big Data

BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities

BitSail, an open‑source data integration engine from ByteDance, provides a unified solution for batch, streaming, full‑load, and incremental data synchronization across heterogeneous sources, detailing its background, technical evolution, architecture, low‑cost co‑building features, compatibility strategies, and future roadmap.

CDCData IntegrationFlink
0 likes · 18 min read
BitSail: ByteDance’s Open‑Source Unified Data Integration Engine – Architecture, Evolution, and Capabilities
IT Services Circle
IT Services Circle
Oct 26, 2022 · Databases

Debezium: Open‑Source Change Data Capture Platform – Overview, Architecture, Use Cases, and Installation Guide

This article introduces Debezium, an open‑source low‑latency change data capture platform that streams database row changes via Kafka, explains its architecture and common scenarios such as cache invalidation and CQRS, and provides step‑by‑step Docker commands to install ZooKeeper, Kafka, MySQL and the Debezium connector.

CDCData IntegrationDebezium
0 likes · 15 min read
Debezium: Open‑Source Change Data Capture Platform – Overview, Architecture, Use Cases, and Installation Guide
DataFunSummit
DataFunSummit
Oct 21, 2022 · Big Data

Exploring Real‑Time Data Lake Practices at Xiaohongshu Using Apache Iceberg

This article details Xiaohongshu's data platform architecture and three real‑time lake initiatives—log ingestion, CDC ingestion, and lake analysis—showcasing how Apache Iceberg, Flink, and custom shuffling algorithms solve small‑file and cross‑cloud challenges while enabling schema evolution and future multi‑cloud optimizations.

Apache IcebergBig DataCDC
0 likes · 16 min read
Exploring Real‑Time Data Lake Practices at Xiaohongshu Using Apache Iceberg
Alibaba Cloud Native
Alibaba Cloud Native
Sep 29, 2022 · Cloud Native

Why Use RocketMQ Connect for Scalable Data Pipelines?

This article explains the challenges of point‑to‑point data sync, introduces RocketMQ Connect as a cloud‑native solution that decouples upstream and downstream, details its architecture, connectors, REST API, metrics, deployment modes, and provides a step‑by‑step guide to building custom connectors for use cases such as CDC, data lakes, and system migration.

CDCCloud NativeConnector
0 likes · 19 min read
Why Use RocketMQ Connect for Scalable Data Pipelines?
DataFunTalk
DataFunTalk
Aug 6, 2022 · Big Data

Exploring Real‑Time Data Lake Practices at Xiaohongshu Using Apache Iceberg

This article details Xiaohongshu's data platform engineering, describing how Apache Iceberg is leveraged for real‑time data lake ingestion, CDC pipelines, multi‑cloud storage, small‑file mitigation, schema evolution, and future plans across storage, compute, and management within a big‑data ecosystem.

Apache IcebergCDCFlink
0 likes · 16 min read
Exploring Real‑Time Data Lake Practices at Xiaohongshu Using Apache Iceberg
Efficient Ops
Efficient Ops
Jul 19, 2022 · Databases

How CDC Powers Real-Time Analytics Without Overloading Your Database

This article introduces the practice of Change Data Capture (CDC), explaining how capturing only data changes can feed downstream systems and data warehouses in near real‑time, reducing load on the source database, improving reporting latency, and supporting scalable, reliable analytics pipelines.

CDCChange Data CaptureData Replication
0 likes · 9 min read
How CDC Powers Real-Time Analytics Without Overloading Your Database
Alibaba Cloud Native
Alibaba Cloud Native
Jul 17, 2022 · Cloud Native

Build Real-Time CDC Pipelines on Alibaba Cloud EventBridge with DTS

This article explains Change Data Capture (CDC) concepts, compares open‑source CDC tools, and shows how to leverage Alibaba Cloud EventBridge and DTS to build real‑time CDC pipelines, covering setup steps, event‑bus vs event‑stream choices, best‑practice scenarios such as CQRS, microservice decoupling, database backup, and SQL auditing.

CDCCloud NativeDTS
0 likes · 12 min read
Build Real-Time CDC Pipelines on Alibaba Cloud EventBridge with DTS
Efficient Ops
Efficient Ops
Jul 6, 2022 · Databases

How DataBus Enables Real-Time, Scalable Database Synchronization for Oracle Migration

DataBus is a real‑time data synchronization framework designed to support Oracle de‑commissioning, micro‑service migration, and heterogeneous storage engines by providing high‑availability CDC, flexible data pipelines, and seamless full‑to‑incremental migration across multiple source and target databases.

CDCData synchronizationDatabase Migration
0 likes · 19 min read
How DataBus Enables Real-Time, Scalable Database Synchronization for Oracle Migration
Bilibili Tech
Bilibili Tech
Jun 10, 2022 · Big Data

Incremental Data Lake Design and Hudi Core Optimizations with Flink

The article describes how combining Apache Flink with Hudi enables an incremental data lake that delivers near‑real‑time analytics by switching to merge‑on‑read, fixing log handling bugs, improving compaction planning, and refactoring table‑service scheduling, while showcasing use cases such as CDC ingestion, data quality control, and real‑time materialized views, and outlines future enhancements like optimistic concurrency and unified schema evolution.

Apache HudiCDCCompaction Optimization
0 likes · 21 min read
Incremental Data Lake Design and Hudi Core Optimizations with Flink
IT Architects Alliance
IT Architects Alliance
Jun 7, 2022 · Databases

Introduction to Change Data Capture (CDC) Practices

This article introduces the concept and practice of Change Data Capture (CDC), explaining how it captures database changes to provide real‑time incremental data for analytics and reporting without impacting source performance, and outlines modern CDC methods, challenges, and production‑ready system requirements.

CDCChange Data CaptureData Integration
0 likes · 8 min read
Introduction to Change Data Capture (CDC) Practices
Top Architect
Top Architect
Jun 7, 2022 · Databases

An Introduction to Change Data Capture (CDC) Practices and Modern Approaches

This article introduces the concept of Change Data Capture (CDC), explains why traditional batch reporting strains resources, describes how CDC captures only data changes to keep source databases performant, and outlines modern CDC architectures, production‑ready considerations, and best‑practice guidelines for building reliable data pipelines.

CDCChange Data CaptureData Integration
0 likes · 16 min read
An Introduction to Change Data Capture (CDC) Practices and Modern Approaches
DataFunTalk
DataFunTalk
May 24, 2022 · Big Data

Integrating Apache Flink with Apache Hudi: From Data Warehouse to Data Lake

This article explains how Apache Flink integrates with Apache Hudi to enable real‑time data lake ingestion, covering the evolution from traditional data warehouses to data lakes, Hudi’s core concepts such as timeline and file grouping, copy‑on‑write vs merge‑on‑read modes, and Flink’s CDC‑based ETL pipeline.

Big DataCDCData Lake
0 likes · 18 min read
Integrating Apache Flink with Apache Hudi: From Data Warehouse to Data Lake
Big Data Technology Architecture
Big Data Technology Architecture
May 22, 2022 · Big Data

Delta Lake Overview, File Structure, Metadata, and Its Integration with Alibaba Cloud EMR, DLF, G‑SCD and CDC Solutions

This article introduces Delta Lake as an open‑source storage layer for lake‑house architectures, explains its key features, file and metadata structures, and details how Alibaba Cloud EMR and Data Lake Formation integrate and extend Delta Lake with advanced capabilities such as G‑SCD, CDC, performance optimizations, and future roadmap.

CDCDLFDelta Lake
0 likes · 10 min read
Delta Lake Overview, File Structure, Metadata, and Its Integration with Alibaba Cloud EMR, DLF, G‑SCD and CDC Solutions
Alibaba Cloud Developer
Alibaba Cloud Developer
May 13, 2022 · Big Data

Unlocking Delta Lake: Key Features, Architecture, and EMR Integration

Delta Lake, an open‑source storage layer from Databricks, provides ACID transactions, data versioning, schema evolution, and unified batch‑stream processing, with a detailed file structure and metadata mechanism, while Alibaba Cloud EMR enhances it with advanced DML, performance optimizations, deep DLF integration, and solutions for G‑SCD and CDC.

CDCDLFData Lakehouse
0 likes · 11 min read
Unlocking Delta Lake: Key Features, Architecture, and EMR Integration
IT Architects Alliance
IT Architects Alliance
May 11, 2022 · Databases

How Change Data Capture Enables Real‑Time Analytics Without Overloading Your Database

The article explains the fundamentals of Change Data Capture (CDC), describing how capturing DML changes from relational databases like MySQL or PostgreSQL can provide incremental, near‑real‑time data for analytics and reporting while preserving source performance, and outlines modern CDC architectures, transaction‑log based extraction, and production‑ready design considerations.

CDCChange Data CaptureDatabase Replication
0 likes · 9 min read
How Change Data Capture Enables Real‑Time Analytics Without Overloading Your Database
Top Architect
Top Architect
May 11, 2022 · Databases

An Introduction to Change Data Capture (CDC) Practices

This article introduces the concept and practice of Change Data Capture (CDC), explaining why CDC is needed for real‑time analytics, how it works by capturing DML changes, modern approaches using transaction logs, and key considerations for building a production‑ready CDC system.

CDCChange Data CaptureData Integration
0 likes · 8 min read
An Introduction to Change Data Capture (CDC) Practices
Shopee Tech Team
Shopee Tech Team
Mar 17, 2022 · Backend Development

Real-time Checking System for Data Consistency in Microservices

Shopee’s Real‑time Checking System provides configurable, non‑intrusive data consistency verification for micro‑services by capturing change events via CDC, streaming them through Kafka, applying flexible rules and expressions, and instantly alerting mismatches, delivering second‑level detection while scaling to tens of thousands of checks per second.

CDCData ConsistencyRedis
0 likes · 20 min read
Real-time Checking System for Data Consistency in Microservices
Volcano Engine Developer Services
Volcano Engine Developer Services
Feb 16, 2022 · Big Data

ByteDance’s Journey to a Unified Data Lake with Flink and Hudi

This article recounts ByteDance’s evolution from batch‑only Flink pipelines to a unified data‑lake integration platform, detailing the three integration modes, challenges with Spark‑based CDC, the decision to adopt Hudi over Iceberg, and how Hudi’s indexing and Merge‑On‑Read formats enable near‑real‑time analytics at massive scale.

CDCFlinkHudi
0 likes · 10 min read
ByteDance’s Journey to a Unified Data Lake with Flink and Hudi
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 16, 2022 · Big Data

Using Flink CDC to Capture MySQL Changes and Sync Them to ClickHouse

This article introduces Change Data Capture (CDC), compares query‑based and log‑based approaches, explains Debezium and ClickHouse, and provides detailed Flink CDC and Flink SQL CDC examples—including Java source code, custom deserialization schema, ClickHouse sink implementation, and required Maven dependencies—to synchronize MySQL data into ClickHouse in real time.

Big DataCDCClickHouse
0 likes · 17 min read
Using Flink CDC to Capture MySQL Changes and Sync Them to ClickHouse
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 22, 2021 · Big Data

Using Flink CDC to Capture MySQL Changes and Sink Them into ClickHouse

This article explains Change Data Capture (CDC), compares query‑based and log‑based approaches, introduces Debezium and ClickHouse, and provides step‑by‑step Flink CDC and Flink SQL CDC examples—including Java source, deserialization, sink code and required Maven dependencies—to stream MySQL binlog changes into ClickHouse for real‑time analytics.

Big DataCDCClickHouse
0 likes · 14 min read
Using Flink CDC to Capture MySQL Changes and Sink Them into ClickHouse