This article explains how to replace daily offline MySQL‑to‑Hive sync with a CDC pipeline using Alibaba’s Canal to capture binlog events, Kafka for transport, and LinkedIn’s Camus (via a custom writer) to load data into Hive, detailing configuration and deployment steps.
This article details how a high‑traffic e‑commerce platform migrated from a single MySQL instance to a sharded architecture using Sharding‑JDBC, added Elasticsearch‑HBase for multi‑dimensional queries, and implemented zero‑downtime data migration and scaling strategies.
This guide provides a step‑by‑step tutorial on downloading, installing, configuring, and starting Alibaba Canal and its adapter to achieve real‑time incremental data synchronization from MySQL binlog to destinations such as Kafka, including code snippets and configuration details.
This guide explains what Canal is, its key features and limitations, the underlying binlog replication principle, and provides detailed, step‑by‑step instructions for downloading, configuring, and launching both Canal Server and Canal Adapter to achieve high‑performance real‑time data synchronization.
This article introduces four change‑data‑capture solutions—Canal, Maxwell, Databus, and Alibaba Cloud DTS—explaining their principles, processing steps, key features, and advantages for real‑time data replication and synchronization across diverse database systems in modern data engineering environments.
This article explains the MySQL binary log (binlog), its structure and contents, the three main uses—point‑in‑time recovery, master‑slave replication, and audit logging—followed by a detailed walkthrough of the replication process, an overview of Alibaba's Canal mechanism, and extensive code examples illustrating binlog parsing and event handling.
This article introduces several MySQL binlog-based change data capture solutions—including Canal, Maxwell, Databus, and Alibaba Cloud's Data Transmission Service—explaining their principles, architecture, features, and usage considerations for incremental data subscription and processing.
This article walks through the end‑to‑end setup of a small‑scale data platform using Alibaba's Canal for MySQL binlog capture, covering the installation and configuration of MySQL, Zookeeper, Kafka, and Canal itself, and demonstrates real‑time change capture with sample DML operations.
This article provides a step‑by‑step tutorial on building a small‑scale data platform by installing MySQL, Zookeeper, Kafka and the open‑source Canal middleware, configuring Canal to capture MySQL binlog events, and forwarding the structured data to Kafka for downstream processing.
This article details a step‑by‑step guide for deploying Canal to capture MySQL binlog events, configure HA with ZooKeeper, design a client that parses binlog into JSON, asynchronously acknowledges messages, archive data to local files for batch upload to HDFS, and monitor latency for alerts.
This article explains how to use Alibaba's Canal to capture MySQL binlog changes in real time, covering its underlying protocol, component architecture, HA design with ZooKeeper, configuration steps, deployment examples, and a detailed comparison with alternative tools such as Maxwell and mysql_streamer.
This guide explains how to download, install, and configure Alibaba Canal—including extracting the package, setting up canal.properties, instance.properties, and instance.xml files, and tuning key parameters—to enable reliable MySQL binlog capture for big‑data pipelines.
This guide introduces the canal‑kafka integration package, outlines its constraints, and provides a step‑by‑step tutorial with Maven dependencies and Java source code for a SimpleCanalClient, a Kafka producer, and a server class, enabling a functional demo of Canal to Kafka data streaming.
This article introduces Alibaba's Canal project, explaining its background, architecture, and how it parses MySQL binlog for real‑time data subscription, and provides a complete Java client example with code snippets to demonstrate connecting, fetching, and processing binlog events.
This article describes a complete solution that uses Alibaba's Canal for real‑time MySQL binlog collection, Kafka for transport, and a customized Camus pipeline to load and merge binlog data into Hive, addressing performance, consistency, and delete‑event challenges in large‑scale data warehousing.
The article explains how DBus ingests data from relational databases and log sources by detailing its extractor, incremental conversion, and full‑pull modules, the use of Canal and Kafka, rule‑based log structuring, the unified UMS message format, and heartbeat monitoring for reliability.
Canal, an open‑source Alibaba project, mimics MySQL slave behavior to subscribe to binlog events, parses them, and supports both standalone and ZooKeeper‑coordinated cluster deployments, offering flexible state storage, message processing pipelines, and integration options such as TCP, Kafka, and RocketMQ for real‑time data synchronization.
YouZan’s order data synchronization moves changes from MySQL through Canal‑parsed binlogs into a message queue, then uses sequential SeqNo‑based optimistic locking and HBase’s column‑version timestamps to guarantee ordering for both single‑ and multi‑table updates, while a Logstash‑style configurable pipeline feeds ES for search and HBase for detail queries, eliminating ordered‑queue bottlenecks and ensuring high‑throughput consistency.
This article introduces the concept of data heterogeneity in large-scale systems, explains how sharding creates query challenges, describes query‑dimension and aggregation heterogeneity, and details implementation techniques such as subscribing to MySQL binlog and using Alibaba’s Canal for reliable data synchronization and storage.
The article presents a CDC solution that combines Alibaba’s Canal for real‑time MySQL binlog capture into Kafka with LinkedIn’s Camus for hourly Kafka‑to‑Hive loading, then merges snapshots and incremental binlog data to accurately and efficiently rebuild ODS tables, supporting sharding and delete events.
This article explains the design and implementation of the DWS platform, detailing its log‑driven architecture with Dbus, Wormhole, and Swifts, the technical choices behind real‑time data extraction, transformation, and delivery, and real‑world use cases in finance.
