BestHub
Sign in
Tagged articles

Data synchronization

234 articles · Page 3 of 3
Java High-Performance Architecture
Java High-Performance Architecture
Mar 4, 2019 · Databases

Data Heterogeneity Explained: Sharding, Query Dimensions & MySQL Binlog with Canal

This article introduces the concept of data heterogeneity in large-scale systems, explains how sharding creates query challenges, describes query‑dimension and aggregation heterogeneity, and details implementation techniques such as subscribing to MySQL binlog and using Alibaba’s Canal for reliable data synchronization and storage.

CanalData synchronizationMySQL
0 likes · 7 min read
Data Heterogeneity Explained: Sharding, Query Dimensions & MySQL Binlog with Canal
dbaplus Community
dbaplus Community
Jan 23, 2019 · Big Data

How Zhihu Built a Scalable Data‑Sync Platform with Sqoop and DataX

This article explains Zhihu's journey from ad‑hoc MySQL‑Hive sync using Oozie + Sqoop to a unified, platform‑based data synchronization service that now handles thousands of tables, over 10 TB daily, with load‑aware scheduling, incremental pulls, schema change handling, and tight integration with their offline job scheduler.

Big DataData synchronizationDataX
0 likes · 14 min read
How Zhihu Built a Scalable Data‑Sync Platform with Sqoop and DataX
HomeTech
HomeTech
Jan 9, 2019 · Big Data

Leveraging Apache Ignite as a Memory‑Centric Data Platform to Replace MySQL for High‑Performance Data Synchronization

This article explains how Apache Ignite, a memory‑centric data platform with strong SQL support, can replace costly MySQL join queries, reduce data‑sync time from over 20 minutes to 15 minutes, improve performance fourfold, and outlines architectural changes, SQL advantages, current limitations, and future scaling plans.

Apache IgniteData synchronizationDistributed Queries
0 likes · 8 min read
Leveraging Apache Ignite as a Memory‑Centric Data Platform to Replace MySQL for High‑Performance Data Synchronization
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
Dec 17, 2018 · Databases

Resolving Data Inconsistency Between Redis Cache and MySQL Database

In high‑concurrency scenarios, using Redis as a buffer can improve performance, but without proper strategies such as delayed double‑delete, cache expiration, or asynchronous binlog‑driven updates, inconsistencies between Redis and MySQL may arise, so this article explains the causes and presents four practical solutions.

Cache ConsistencyData synchronizationMySQL
0 likes · 6 min read
Resolving Data Inconsistency Between Redis Cache and MySQL Database
Youzan Coder
Youzan Coder
Dec 7, 2018 · Backend Development

How Youzan Scaled Order Management: Sharding, Elasticsearch, and HBase

The article details Youzan's three‑stage evolution of its order management system—from database sharding to Elasticsearch‑based cross‑table search and finally HBase for fast detail assembly—while addressing data sync, real‑time consistency, and idempotency strategies.

Backend ArchitectureData synchronizationSharding
0 likes · 9 min read
How Youzan Scaled Order Management: Sharding, Elasticsearch, and HBase
21CTO
21CTO
Dec 3, 2018 · Operations

How JD Daojia Scaled Its Elasticsearch Cluster to Billions of Docs: Lessons and Pitfalls

This article details JD Daojia's order center Elasticsearch architecture evolution—from a chaotic initial deployment to a real‑time dual‑cluster backup—covering scaling strategies, data synchronization methods, and the practical pitfalls encountered along the way.

Cluster ArchitectureData synchronizationElasticsearch
0 likes · 14 min read
How JD Daojia Scaled Its Elasticsearch Cluster to Billions of Docs: Lessons and Pitfalls
JD Tech
JD Tech
Dec 3, 2018 · Backend Development

Evolution of JD.com Order Center Elasticsearch Cluster Architecture

This article details how JD.com’s order center migrated its Elasticsearch cluster through multiple stages—from an initial unoptimized deployment to a real‑time dual‑cluster backup solution—addressing scalability, reliability, shard tuning, version upgrades, and data synchronization strategies to support billions of documents and hundreds of millions of daily queries.

Cluster ArchitectureData synchronizationElasticsearch
0 likes · 13 min read
Evolution of JD.com Order Center Elasticsearch Cluster Architecture
Dada Group Technology
Dada Group Technology
Nov 30, 2018 · Big Data

Evolution of JD Daojia Order Center Elasticsearch Cluster: Architecture, Scaling, and Lessons Learned

This article details how JD Daojia's order center migrated from MySQL to a multi‑stage Elasticsearch cluster—covering initial deployment, isolation, replica tuning, primary‑secondary setup, real‑time dual‑cluster upgrades, data synchronization methods, and key pitfalls—to achieve massive scalability, high availability, and performance for billions of orders.

Cluster ArchitectureData synchronizationElasticsearch
0 likes · 13 min read
Evolution of JD Daojia Order Center Elasticsearch Cluster: Architecture, Scaling, and Lessons Learned
System Architect Go
System Architect Go
Aug 4, 2018 · Databases

Synchronizing MySQL Data to Elasticsearch: Methods and Practices

This article reviews various approaches for keeping MySQL data in sync with Elasticsearch, including direct business‑layer hooks, independent synchronization via plugins or custom scripts, and real‑time binlog subscription using tools like zongji, while discussing their advantages, drawbacks, and implementation details.

Data synchronizationElasticsearchMySQL
0 likes · 4 min read
Synchronizing MySQL Data to Elasticsearch: Methods and Practices
Youzan Coder
Youzan Coder
Jul 20, 2018 · Big Data

How Youzan Built a Scalable Big Data Development Platform (DP)

This article details the design, architecture, and operational experience of Youzan's Data Platform (DP), covering its scheduling, data‑sync, service, and monitoring modules, the custom Airflow‑based task scheduler, current production metrics, supported task types, and future improvement plans.

AirflowBig DataData synchronization
0 likes · 12 min read
How Youzan Built a Scalable Big Data Development Platform (DP)
ITPUB
ITPUB
Jun 5, 2018 · Operations

How Meituan Achieved Near‑Zero Downtime for Its Account Service

This article details Meituan's practical approaches to boosting account service reliability, covering MTBF/MTTR metrics, business‑level monitoring, flexible availability with circuit‑breaker patterns, cross‑region active‑active deployment, data synchronization techniques, and the measurable performance gains achieved.

Active-ActiveData synchronizationHigh Availability
0 likes · 13 min read
How Meituan Achieved Near‑Zero Downtime for Its Account Service
Meituan Technology Team
Meituan Technology Team
May 31, 2018 · Operations

High‑Availability Practices for Account Services at Meituan/Dianping

Meituan/Dianping ensures its critical account service stays online by combining real‑time business monitoring, circuit‑breaker‑driven graceful degradation, and active‑active cross‑region deployment with isolated dependencies, versioned data sync, and automated cache updates, dramatically extending MTBF while cutting MTTR and latency.

Data synchronizationFault ToleranceHigh Availability
0 likes · 13 min read
High‑Availability Practices for Account Services at Meituan/Dianping
JD Tech
JD Tech
May 31, 2018 · Backend Development

Design and Architecture of a Unified MySQL Data Synchronization Platform

This article details the design of a unified MySQL data synchronization platform that consolidates offline sync, real‑time subscription, and real‑time sync into BatchJob, StreamJob, and PieJob abstractions, describing task implementations, cluster architecture, high‑availability mechanisms, and evolution challenges such as file loss and metadata handling.

Backend ArchitectureBatch ProcessingData synchronization
0 likes · 10 min read
Design and Architecture of a Unified MySQL Data Synchronization Platform
Qunar Tech Salon
Qunar Tech Salon
Nov 28, 2017 · Backend Development

Optimizing the TTS Pricing Engine: From Monolithic V1 to Scalable V2 Architecture

This article details the evolution of Qunar's TTS pricing engine from a monolithic V1 implementation with dual-level caching and blocking HTTP interfaces to a scalable V2 architecture featuring service separation, asynchronous processing, CQRS-based rule storage, canal synchronization, Solr and Redis indexing, and Kafka-driven data pipelines.

Data synchronizationPricing EngineRedis
0 likes · 10 min read
Optimizing the TTS Pricing Engine: From Monolithic V1 to Scalable V2 Architecture
Tongcheng Travel Technology Center
Tongcheng Travel Technology Center
Nov 23, 2017 · Backend Development

Design and Implementation of a Search Open Platform for Rapid Interface Provision

The article describes the requirements, architecture, data‑sync strategy, monitoring, and operational workflow of a search open platform that enables fast, zero‑code creation of searchable interfaces, supporting real‑time indexing, customizable ranking, and extensible backend services.

Data synchronizationIndexingMonitoring
0 likes · 12 min read
Design and Implementation of a Search Open Platform for Rapid Interface Provision
Hujiang Technology
Hujiang Technology
Oct 25, 2017 · Backend Development

Technical Case Study of Migrating a Transaction System from Windows/.NET/SQL Server to Linux/Java/MySQL

This article presents a comprehensive technical case study of a large‑scale migration that replaces a Windows/.NET/SQL Server stack with a Linux/Java/MySQL architecture, detailing system decomposition, order service redesign, code conversion scripts, data synchronization challenges, testing strategies, deployment and rollback procedures.

Data synchronizationMigrationsystem architecture
0 likes · 21 min read
Technical Case Study of Migrating a Transaction System from Windows/.NET/SQL Server to Linux/Java/MySQL
CoolHome R&D Department
CoolHome R&D Department
Sep 27, 2017 · Frontend Development

How to Build Reactive Data Synchronization Between Raw and Intermediate Models

This article explains why an intermediate data structure is needed for 3D rendering in a React‑like Virtual DOM environment, how to keep it reactively synchronized with raw graph data using dependency analysis, proxies, and conversion functions, and the performance benefits of minimal partial updates.

Data synchronizationFrontendReactive Programming
0 likes · 12 min read
How to Build Reactive Data Synchronization Between Raw and Intermediate Models
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Jul 17, 2017 · Big Data

Mastering Data Sync, Real‑Time Processing, and Scalable Storage for Modern Systems

This article explores practical techniques for synchronizing heterogeneous data sources, performing batch and incremental analytics with Hadoop and Spark, designing low‑latency real‑time computation pipelines, implementing push notifications, and choosing appropriate storage solutions—from in‑memory caches to distributed databases—while addressing performance, reliability, and scalability challenges.

Big DataData synchronizationDatabases
0 likes · 25 min read
Mastering Data Sync, Real‑Time Processing, and Scalable Storage for Modern Systems
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 15, 2017 · Backend Development

Taobao’s Secret to Billions of API Calls: High‑Performance Gateway & Reliable Messaging

Taobao’s Open Platform sustains hundreds of billions of daily API calls and messages by employing a pipeline‑based high‑performance API gateway with multi‑level caching, asynchronous processing, granular traffic control, a highly reliable push‑pull messaging system, and a zero‑loss data‑sync service that dynamically balances resources during massive traffic spikes.

API GatewayBackend ArchitectureData synchronization
0 likes · 18 min read
Taobao’s Secret to Billions of API Calls: High‑Performance Gateway & Reliable Messaging
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Jan 5, 2017 · Databases

Mongosync: Fast Full and Real-Time MongoDB Sync with Benchmarks

This article introduces Mongosync, a Qihoo 360‑developed tool for transferring data between MongoDB replica sets, detailing its background, supported features such as full‑data and real‑time oplog synchronization, version compatibility, multithreaded bulk writes, configurable options, performance test results, and future development plans.

Data synchronizationMongoDBMongosync
0 likes · 7 min read
Mongosync: Fast Full and Real-Time MongoDB Sync with Benchmarks
Qunar Tech Salon
Qunar Tech Salon
Nov 24, 2016 · Backend Development

Practical Application Scenarios of Queues in System Design

This article explains how queues—such as message queues, task queues, and data‑bus queues—are applied in backend systems for asynchronous processing, system decoupling, data synchronization, traffic shaping, buffering, and other architectural patterns to improve scalability and reliability.

Data synchronizationSystem DecouplingTask queue
0 likes · 10 min read
Practical Application Scenarios of Queues in System Design
ITPUB
ITPUB
Oct 21, 2016 · Databases

How I Reclaimed 200 GB by Cleaning SQL Server Backups and Syncing Data

A developer discovers a nearly full 1 TB server disk caused by large SQL Server backup files, learns the backup strategy, removes old backups while keeping monthly snapshots, manually syncs production data to a test server, and reflects on backup file formats and storage optimization.

DBAData synchronizationSQL Server
0 likes · 7 min read
How I Reclaimed 200 GB by Cleaning SQL Server Backups and Syncing Data
ITFLY8 Architecture Home
ITFLY8 Architecture Home
Oct 16, 2016 · Big Data

Mastering Data Sync, Real-Time Analytics, and Scalable Storage for Modern Systems

This article explains how to design and implement heterogeneous data synchronization, leverage batch and stream processing frameworks like Hadoop and Storm for large‑scale analysis, and choose appropriate storage solutions—from in‑memory databases to distributed column‑family stores—while addressing performance, reliability, and monitoring in complex distributed environments.

Big DataData synchronizationDatabases
0 likes · 26 min read
Mastering Data Sync, Real-Time Analytics, and Scalable Storage for Modern Systems
Efficient Ops
Efficient Ops
Aug 1, 2016 · Operations

How Tencent Shifted 70M Users During Tianjin Explosion – A Multi‑Active Ops Playbook

This article details how Tencent's operations team orchestrated a seamless, zero‑impact migration of over 70 million users across three data centers during the 2015 Tianjin explosion, highlighting the four key capabilities—distribution, scheduling, data synchronization, and automated operations—that enabled multi‑active disaster recovery at massive scale.

Data synchronizationDisaster RecoveryLoad Balancing
0 likes · 22 min read
How Tencent Shifted 70M Users During Tianjin Explosion – A Multi‑Active Ops Playbook
Efficient Ops
Efficient Ops
Jul 25, 2016 · Operations

How We Overcame Real‑World Challenges in a Large‑Scale Oracle Database Cutover

This article recounts a seven‑year‑old Oracle 10g database migration, detailing project background, team turmoil, topology redesign, security constraints, data‑sync strategies, custom tools, high‑fidelity testing, unexpected failures, and the lessons learned for reliable operations.

Data synchronizationDatabase MigrationOperations
0 likes · 14 min read
How We Overcame Real‑World Challenges in a Large‑Scale Oracle Database Cutover
Big Data and Microservices
Big Data and Microservices
Apr 12, 2016 · Backend Development

WeChat’s Backend Journey: From Zero to Billions with Scalable Architecture

This article chronicles how WeChat’s backend evolved from a simple messaging prototype to a globally distributed, multi‑data‑center system, detailing its message model, unified sync protocol, three‑layer architecture, platformization, disaster‑recovery design, performance tuning, and emerging resource‑scheduling challenges.

Backend ArchitectureData synchronizationDisaster Recovery
0 likes · 30 min read
WeChat’s Backend Journey: From Zero to Billions with Scalable Architecture
21CTO
21CTO
Mar 13, 2016 · Mobile Development

How to Build Robust Offline Support for Mobile Apps

This article explains why offline capability is essential for mobile apps, outlines key requirements such as error handling and guest mode, and details practical solutions including local caching strategies, queueing requests, data synchronization methods, and recommended libraries.

Data synchronizationlocal queuemobile caching
0 likes · 14 min read
How to Build Robust Offline Support for Mobile Apps
High Availability Architecture
High Availability Architecture
Feb 26, 2016 · Databases

Weibo Database Architecture: Evolution, Optimization, and Design Practices

This article details the evolution of Weibo's database platform—from its early single‑master design through rapid scaling, vertical and horizontal sharding, automation, Redis enhancements, custom middleware, and future plans—offering practical insights into high‑availability, high‑performance data engineering for large‑scale social media services.

AutomationData synchronizationDatabase Architecture
0 likes · 23 min read
Weibo Database Architecture: Evolution, Optimization, and Design Practices
Qunar Tech Salon
Qunar Tech Salon
Jan 16, 2016 · Backend Development

From Zero to One: The Evolution of WeChat’s Backend System Architecture

This article chronicles the two‑month development of WeChat’s backend from its inception, detailing the design of its message model, data‑sync protocol, three‑tier architecture, asynchronous queues, rapid scaling, platformization, multi‑data‑center deployment, disaster‑recovery strategies, performance optimizations, security hardening, and emerging resource‑scheduling challenges.

Data synchronizationOperationsWeChat
0 likes · 28 min read
From Zero to One: The Evolution of WeChat’s Backend System Architecture
Architect
Architect
Dec 21, 2015 · Databases

Alibaba's Massive Data Architecture: Sharding, Multi‑Data‑Center Synchronization, and Caching Strategies

The article describes how Alibaba scaled its massive data platform by horizontally sharding Oracle tables to MySQL, employing multi‑data‑center synchronization tools such as Erosa, Eromanga, and Otter, and designing multi‑level caching and proxy solutions like Cobar to ensure high availability and performance.

AlibabaData synchronizationcaching
0 likes · 16 min read
Alibaba's Massive Data Architecture: Sharding, Multi‑Data‑Center Synchronization, and Caching Strategies
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Dec 18, 2015 · Cloud Computing

Alibaba's Evolution to Geo‑Distributed Multi‑Active Architecture: Design, Challenges, and Lessons

The article details Alibaba's three‑stage evolution from same‑city active‑active to two‑site three‑center disaster recovery and finally to geo‑distributed multi‑active architecture, explaining its motivations, technical challenges such as latency and data consistency, the solutions implemented, and the resulting scalability and fault‑tolerance benefits.

AlibabaData synchronizationMulti-Active
0 likes · 16 min read
Alibaba's Evolution to Geo‑Distributed Multi‑Active Architecture: Design, Challenges, and Lessons