EPX: Real-Time MySQL Change Capture and Kafka Sync Architecture

EPX is a high‑availability, high‑performance data pipeline that captures MySQL binlog changes in real time and streams them to Kafka for downstream consumption.

It implements MySQL master‑slave replication principles: the slave creates I/O and SQL threads, reads binlog from the master, writes to a relay log, and replays statements to keep data consistent.

EPX mimics a MySQL slave, sends a dump request to the master, receives binary log streams, parses and filters them, transforms the events into a unified JSON format, and writes them to Kafka.

The architecture consists of ConfigManager, TaskManager, EventParse (based on Alibaba Canal), EventHandler, KafkaWriter, and PositionManager, all coordinated via Zookeeper for load balancing and offset tracking.

Key functions include real‑time change monitoring (INSERT, UPDATE, DELETE), offline statistics and backup to Hive, database migration support, latency and traffic monitoring, and customizable anomaly alerts.

High‑availability is achieved through multiple worker clusters, multi‑data‑center deployment, automatic load balancing, and fast failover, aiming for five‑nine availability.

EPX is currently used across 32 departments at the company, supporting search, recommendation, data warehousing, real‑time metrics, and critical field change subscriptions, with over 1,131 tables monitored.

Future plans focus on further latency reduction, additional Kafka outputs, MySQL master‑slave failover, and continued architectural optimizations.