To meet growing e‑commerce search demands, the team replaced a MySQL‑based intermediate table with a real‑time binlog‑driven pipeline that streams changes through Kafka into Elasticsearch, detailing design choices, ordering and completeness guarantees, custom modules, and monitoring for sub‑second sync latency.
This article details JD.com’s order center journey from a simple Elasticsearch deployment to a highly available, dual‑cluster architecture, covering isolation, replica tuning, hot‑cold data separation, version upgrades, and practical lessons on pagination, field data, and doc values.
This article explains the financial‑industry driven requirements for real‑time data sync, describes the TDSQL‑MULTISRCSYNC architecture—including producer, store, and consumer components—and details core designs such as row‑hash concurrency, idempotent binlog handling, and a lock‑based ordering mechanism that ensure high throughput and consistency.
This article explains the concepts of unitization and multi‑active data centers, analyzes the drawbacks of single‑IDC deployments, and provides a detailed guide on implementing MySQL binlog synchronization while addressing common challenges such as data loops, duplicate inserts, unique‑index conflicts, and DDL handling, including practical solutions like GTID and auxiliary tables.
This article outlines the CoolJia search service quality system, detailing its architecture, the three‑pronged business testing approach, performance and effect testing, data synchronization tests, and the dedicated debug platform to ensure reliable, high‑performance search operations.
data synchronizationdebug platformperformance testing
The article explains how MongoDB Change Streams, introduced in version 3.6 and expanded in 4.0, enable near‑real‑time subscription to collection, database, and cluster changes, discusses implementation details, options, code examples in the mongo shell and Python, and provides practical testing insights and driver compatibility notes.
This article presents a comprehensive overview of Taobao's cloud computing architecture, detailing system components, various data synchronization methods such as TimeTunnel, Dbsync, and DataX, the scheduling system design, and metadata-driven analysis platforms for performance optimization and monitoring.
YouZan’s order data synchronization moves changes from MySQL through Canal‑parsed binlogs into a message queue, then uses sequential SeqNo‑based optimistic locking and HBase’s column‑version timestamps to guarantee ordering for both single‑ and multi‑table updates, while a Logstash‑style configurable pipeline feeds ES for search and HBase for detail queries, eliminating ordered‑queue bottlenecks and ensuring high‑throughput consistency.
Modern cloud outages reveal that cold‑standby or simple multi‑cloud promises often provide only psychological comfort; achieving true high availability requires active‑active designs with local traffic handling, data partitioning, and low‑latency synchronization, while balancing cost, complexity, and physical distance constraints.
This article introduces the concept of data heterogeneity in large-scale systems, explains how sharding creates query challenges, describes query‑dimension and aggregation heterogeneity, and details implementation techniques such as subscribing to MySQL binlog and using Alibaba’s Canal for reliable data synchronization and storage.
This article explains Zhihu's journey from ad‑hoc MySQL‑Hive sync using Oozie + Sqoop to a unified, platform‑based data synchronization service that now handles thousands of tables, over 10 TB daily, with load‑aware scheduling, incremental pulls, schema change handling, and tight integration with their offline job scheduler.
This article explains how Apache Ignite, a memory‑centric data platform with strong SQL support, can replace costly MySQL join queries, reduce data‑sync time from over 20 minutes to 15 minutes, improve performance fourfold, and outlines architectural changes, SQL advantages, current limitations, and future scaling plans.
In high‑concurrency scenarios, using Redis as a buffer can improve performance, but without proper strategies such as delayed double‑delete, cache expiration, or asynchronous binlog‑driven updates, inconsistencies between Redis and MySQL may arise, so this article explains the causes and presents four practical solutions.
The article details Youzan's three‑stage evolution of its order management system—from database sharding to Elasticsearch‑based cross‑table search and finally HBase for fast detail assembly—while addressing data sync, real‑time consistency, and idempotency strategies.
This article details JD Daojia's order center Elasticsearch architecture evolution—from a chaotic initial deployment to a real‑time dual‑cluster backup—covering scaling strategies, data synchronization methods, and the practical pitfalls encountered along the way.
This article details how JD.com’s order center migrated its Elasticsearch cluster through multiple stages—from an initial unoptimized deployment to a real‑time dual‑cluster backup solution—addressing scalability, reliability, shard tuning, version upgrades, and data synchronization strategies to support billions of documents and hundreds of millions of daily queries.
This article details how JD Daojia's order center migrated from MySQL to a multi‑stage Elasticsearch cluster—covering initial deployment, isolation, replica tuning, primary‑secondary setup, real‑time dual‑cluster upgrades, data synchronization methods, and key pitfalls—to achieve massive scalability, high availability, and performance for billions of orders.
This article reviews various approaches for keeping MySQL data in sync with Elasticsearch, including direct business‑layer hooks, independent synchronization via plugins or custom scripts, and real‑time binlog subscription using tools like zongji, while discussing their advantages, drawbacks, and implementation details.
This article details the design, architecture, and operational experience of Youzan's Data Platform (DP), covering its scheduling, data‑sync, service, and monitoring modules, the custom Airflow‑based task scheduler, current production metrics, supported task types, and future improvement plans.
This article details Meituan's practical approaches to boosting account service reliability, covering MTBF/MTTR metrics, business‑level monitoring, flexible availability with circuit‑breaker patterns, cross‑region active‑active deployment, data synchronization techniques, and the measurable performance gains achieved.
Meituan/Dianping ensures its critical account service stays online by combining real‑time business monitoring, circuit‑breaker‑driven graceful degradation, and active‑active cross‑region deployment with isolated dependencies, versioned data sync, and automated cache updates, dramatically extending MTBF while cutting MTTR and latency.
High Availabilitydata synchronizationfault tolerance
This article details the design of a unified MySQL data synchronization platform that consolidates offline sync, real‑time subscription, and real‑time sync into BatchJob, StreamJob, and PieJob abstractions, describing task implementations, cluster architecture, high‑availability mechanisms, and evolution challenges such as file loss and metadata handling.
This article details the evolution of Qunar's TTS pricing engine from a monolithic V1 implementation with dual-level caching and blocking HTTP interfaces to a scalable V2 architecture featuring service separation, asynchronous processing, CQRS-based rule storage, canal synchronization, Solr and Redis indexing, and Kafka-driven data pipelines.
The article describes the requirements, architecture, data‑sync strategy, monitoring, and operational workflow of a search open platform that enables fast, zero‑code creation of searchable interfaces, supporting real‑time indexing, customizable ranking, and extensible backend services.
This article presents a comprehensive technical case study of a large‑scale migration that replaces a Windows/.NET/SQL Server stack with a Linux/Java/MySQL architecture, detailing system decomposition, order service redesign, code conversion scripts, data synchronization challenges, testing strategies, deployment and rollback procedures.
This article explains why an intermediate data structure is needed for 3D rendering in a React‑like Virtual DOM environment, how to keep it reactively synchronized with raw graph data using dependency analysis, proxies, and conversion functions, and the performance benefits of minimal partial updates.
This article presents the design and implementation of a smooth migration solution that moves Baidu's internal bdrp service from Redis 2.6/2.8 to Redis 3.0 by extending the AOF mechanism, enabling real‑time data synchronization, improving service stability, and supporting future upgrades.
This article explores practical techniques for synchronizing heterogeneous data sources, performing batch and incremental analytics with Hadoop and Spark, designing low‑latency real‑time computation pipelines, implementing push notifications, and choosing appropriate storage solutions—from in‑memory caches to distributed databases—while addressing performance, reliability, and scalability challenges.
Taobao’s Open Platform sustains hundreds of billions of daily API calls and messages by employing a pipeline‑based high‑performance API gateway with multi‑level caching, asynchronous processing, granular traffic control, a highly reliable push‑pull messaging system, and a zero‑loss data‑sync service that dynamically balances resources during massive traffic spikes.
API Gatewaybackend architecturedata synchronization
Learn how to ensure that a user list page instantly reflects edits made on a separate detail page in a WeChat Mini Program by using a lightweight broadcast utility, avoiding full page reloads and improving user experience.
This article introduces Mongosync, a Qihoo 360‑developed tool for transferring data between MongoDB replica sets, detailing its background, supported features such as full‑data and real‑time oplog synchronization, version compatibility, multithreaded bulk writes, configurable options, performance test results, and future development plans.
This article explains how queues—such as message queues, task queues, and data‑bus queues—are applied in backend systems for asynchronous processing, system decoupling, data synchronization, traffic shaping, buffering, and other architectural patterns to improve scalability and reliability.
A developer discovers a nearly full 1 TB server disk caused by large SQL Server backup files, learns the backup strategy, removes old backups while keeping monthly snapshots, manually syncs production data to a test server, and reflects on backup file formats and storage optimization.
This article explains how to design and implement heterogeneous data synchronization, leverage batch and stream processing frameworks like Hadoop and Storm for large‑scale analysis, and choose appropriate storage solutions—from in‑memory databases to distributed column‑family stores—while addressing performance, reliability, and monitoring in complex distributed environments.
This article details how Tencent's operations team orchestrated a seamless, zero‑impact migration of over 70 million users across three data centers during the 2015 Tianjin explosion, highlighting the four key capabilities—distribution, scheduling, data synchronization, and automated operations—that enabled multi‑active disaster recovery at massive scale.
This guide explains how to enable Alibaba Cloud Data Transmission Service (DTS) with the dts-ads-writer plugin to continuously synchronize changes from an RDS for MySQL instance to a real‑time write table in an analytic database, covering prerequisites, configuration, and monitoring steps.
This article chronicles how WeChat’s backend evolved from a simple messaging prototype to a globally distributed, multi‑data‑center system, detailing its message model, unified sync protocol, three‑layer architecture, platformization, disaster‑recovery design, performance tuning, and emerging resource‑scheduling challenges.
This article explains why offline capability is essential for mobile apps, outlines key requirements such as error handling and guest mode, and details practical solutions including local caching strategies, queueing requests, data synchronization methods, and recommended libraries.
This article details the evolution of Weibo's database platform—from its early single‑master design through rapid scaling, vertical and horizontal sharding, automation, Redis enhancements, custom middleware, and future plans—offering practical insights into high‑availability, high‑performance data engineering for large‑scale social media services.
This article chronicles the two‑month development of WeChat’s backend from its inception, detailing the design of its message model, data‑sync protocol, three‑tier architecture, asynchronous queues, rapid scaling, platformization, multi‑data‑center deployment, disaster‑recovery strategies, performance optimizations, security hardening, and emerging resource‑scheduling challenges.
The article describes how Alibaba scaled its massive data platform by horizontally sharding Oracle tables to MySQL, employing multi‑data‑center synchronization tools such as Erosa, Eromanga, and Otter, and designing multi‑level caching and proxy solutions like Cobar to ensure high availability and performance.
The article details Alibaba's three‑stage evolution from same‑city active‑active to two‑site three‑center disaster recovery and finally to geo‑distributed multi‑active architecture, explaining its motivations, technical challenges such as latency and data consistency, the solutions implemented, and the resulting scalability and fault‑tolerance benefits.