Uber tackled the challenge of replicating over 350 PB of data across on‑premise and cloud lakes by redesigning Hadoop Distcp, moving intensive tasks to the Application Master, parallelising copy‑listing and commit phases, and leveraging Uber‑mapper jobs to dramatically cut latency and improve resource efficiency.
This article analyzes the strategic decision of using local data copies or caches versus central middle‑platform services, examining performance, frequency, cost, technical complexity, and organizational impact through the lens of CAP theorem and modern cloud‑native architecture.
Netflix built a distributed write‑ahead log (WAL) system to guarantee data consistency, durability, and high‑throughput across services, tackling challenges like data loss, cross‑store entropy, multi‑partition updates, replication, and reliable retry for real‑time pipelines.
This article explains the design and implementation of Ctrip's Data Replication Center (DRC), a MySQL‑based high‑availability system that solves cross‑region data loop, progress tracking, concurrency, DDL handling, and conflict resolution to achieve low‑latency, reliable data replication for global travel services.
This guide explains how to achieve a zero‑downtime Kafka cluster migration by deploying a new cluster, configuring MirrorMaker 2 for bidirectional replication, gradually switching producers and consumers, monitoring key metrics, and safely decommissioning the old cluster.
This article walks developers through the evolution of distributed system architectures—from single‑machine deployments to master‑slave, same‑city active‑active, and finally true multi‑active setups—explaining core concepts, replication strategies, conflict resolution, fault detection, switch mechanisms, recovery methods, and interview tips for high‑availability design.
This article explains why most companies suffer long recovery times after data loss, introduces DRBD's real‑time block replication as a solution, and provides detailed architecture designs, deployment steps, monitoring scripts, performance tuning, cost analysis, common pitfalls, and future trends for reliable disaster recovery.
The article analyzes high‑availability concepts, from single‑machine risks to multi‑site active‑active architectures, compares cold and hot backup strategies, discusses network latency challenges, and presents Ele.me’s cell‑based, sharding‑driven multi‑region solution with concrete examples, tables, and code snippets.
This article explores the concept of consistency in distributed systems, breaking down various consistency models—including linear, sequential, causal, and eventual—explaining their definitions, practical implications, and how they guide the design of high‑availability architectures and data replication strategies.
This article explains the principles of leader‑based and multi‑leader replication, compares synchronous and asynchronous modes, discusses consistency trade‑offs, conflict resolution strategies, and architectural variations for single‑ and multi‑data‑center deployments.
This tutorial explains the definitions of faults and failures in distributed systems, explores their types and root causes, and presents fault‑tolerance mechanisms such as replication, checkpointing, redundancy, error detection, load balancing, and consensus algorithms to build resilient architectures.
This article explores the principles of data replication, comparing shared memory, shared disk, and non‑shared storage architectures, detailing replication formats, consistency challenges, and various replication strategies such as synchronous, asynchronous, semi‑synchronous, and majority‑based approaches, helping engineers choose the right trade‑offs.
This article explains how RocketMQ achieves high availability and data reliability through its master‑slave broker design, covering synchronous and asynchronous replication, flush strategies, transaction messaging, automatic failover with Dledger, and read‑write separation for load balancing in distributed systems.
This article analyzes the architectural impact of cross‑city multi‑active deployments, focusing on data‑layer design, write latency, sharding strategies, replication topologies, and routing considerations to achieve high availability, performance, and scalability in large‑scale distributed systems.
This article examines the challenges of building multi‑active distributed systems, focusing on the data layer’s role in high availability, write‑latency, sharding, isolation, replication strategies, and routing decisions, and provides concrete architectural patterns and practical guidelines for robust backend design.
The NineData presentation at the 2024 GOPS Global Operations Conference in Shanghai detailed multi‑cloud, multi‑source database architecture trends, showcased their intelligent data management platform, explained data replication principles, DevOps challenges and AI‑enhanced solutions, and highlighted real‑world customer success stories across industries.
This article analyzes the architectural challenges of building large‑scale distributed systems with multi‑active (cross‑city) capabilities, focusing on data‑layer design, write latency, replication models, sharding techniques, and routing impacts to guide reliable, high‑performance infrastructure decisions.
This article analyzes how write latency, write volume, isolation, and data replication strategies influence the design of multi‑active distributed systems, offering practical guidance on sharding, synchronous and asynchronous replication, routing, and architecture selection for high availability and performance across regions.
The article introduces MySQL and PostgreSQL, compares their performance through a benchmark on identical hardware, discusses each system’s strengths, weaknesses, and suitable application scenarios, and provides guidance on choosing the appropriate database for different workloads.
The two‑site three‑center disaster recovery model combines a production site, a same‑city backup, and a remote backup to ensure data integrity and rapid recovery, leveraging Oracle Data Guard for synchronized and asynchronous replication, thereby improving RPO and RTO across various disaster scenarios.
To achieve disaster-recovery and cross-data-center resilience, Didi progressed from a simple proxy double-write scheme to a sophisticated MQ-free multi-active Redis design that uses a dedicated syncer, shard-based loop prevention, op-id replay protection, conflict detection, and incremental AOF durability, ensuring low latency, no data loss, and consistent availability.
This article provides a comprehensive overview of Kafka's architecture and fundamental concepts, covering its overall structure, key components such as brokers, producers, consumers, topics, partitions, replicas, leader‑follower synchronization, offset handling, message storage at both logical and physical layers, as well as producer and consumer workflows, partition assignment strategies, rebalancing, log management, zero‑copy I/O, and reliability mechanisms.
This article explains MySQL's binlog, its role in master‑slave replication, and how businesses can harness it for data heterogeneity, cache synchronization, and task dispatch, illustrating practical middleware designs that transform raw changes into valuable services.
This article explains the concepts of eventual consistency, hinted handoff queues and anti‑entropy in distributed databases, illustrates how they work with XDB Enterprise examples, and shows how AE restores data integrity after node failures or network partitions.
Using a fictional coronavirus carried by a bat, the article illustrates the Gossip protocol’s mechanisms—direct mail, anti-entropy, and epidemic spread—to explain how distributed systems achieve eventual consistency, highlighting advantages, drawbacks, and practical considerations for storage components like Cassandra.
This article details Bilibili's multi‑datacenter offline architecture, explaining the capacity challenges, the chosen scale‑out design, and the implementation of job placement, data replication, routing, versioning, throttling, and traffic analysis to efficiently handle massive batch workloads across geographically distributed clusters.
An in‑depth overview compares Hadoop’s HDFS and the open‑source Ceph object storage, detailing their architectures, replication mechanisms, scalability, strengths, limitations, and real‑world enterprise adoption for handling massive large datasets and unstructured data.
This article presents a step‑by‑step experiment on a Cassandra cluster spanning two data centers, demonstrating how token ownership, data distribution, and fault‑tolerance behave when nodes fail or are removed, and explains the observed owns percentages and replication effects.
This article explains the core concepts of data replication, backup strategies, and disaster recovery, covering RTO/RPO metrics, backup types, copy data management, and the differences between data‑level, application‑level, and business‑level disaster recovery solutions.
This guide outlines the motivations, architectural guidelines, routing strategies, RPC and message replication techniques, storage synchronization methods, and traffic‑switching procedures needed to successfully implement a multi‑active, cross‑region system.
This article explains the concept of data replication, its three-stage process, key principles of compliance, timeliness, and diversity, various replication methods, layered technologies across storage, operating system, and database levels, emerging cloud and big‑data solutions, and heterogeneous use‑case scenarios.
This article describes Bilibili's offline multi‑datacenter architecture, explaining why a scale‑out approach was chosen over scale‑up, and detailing the unit‑based design, job placement, data replication, routing, versioning, bandwidth throttling, traffic analysis, and the operational results and future directions.
The article explains data replication concepts, processes, and technologies across storage hardware, operating system, and database layers, outlines synchronous, asynchronous, and hybrid methods, discusses industry applications, trends such as hardware‑software decoupling, cloud replication, and big‑data real‑time copying, and highlights challenges and future directions.
The article surveys three data‑replication models—master‑slave, multi‑master, and leaderless—explains how they enable scalability and fault‑tolerance, and examines core distributed‑system challenges such as partial node failures, unreliable networks, and unsynchronized clocks, while stressing safety‑liveness trade‑offs and design techniques like quorum and timeouts.
This article introduces the practice of Change Data Capture (CDC), explaining how capturing only data changes can feed downstream systems and data warehouses in near real‑time, reducing load on the source database, improving reporting latency, and supporting scalable, reliable analytics pipelines.
To overcome rapid data growth and on‑premise capacity limits, Bilibili adopted a scale‑out, unit‑based multi‑datacenter architecture that isolates failures, intelligently places jobs, replicates data via an enhanced DistCp service, routes reads with an IP‑aware HDFS router, and throttles cross‑site traffic, enabling stable offline big‑data processing of hundreds of petabytes while preserving throughput.
This article introduces the concept and practice of Change Data Capture (CDC), explaining why CDC is needed for real‑time analytics, how it works by capturing DML changes, modern approaches using transaction logs, and key considerations for building a production‑ready CDC system.
The DTLE 4.22.04.0 release introduces UTF‑32 support, Chinese table name handling, enhanced ColumnMapFrom conversion, expanded SqlFilter capabilities, additional logging, and numerous bug fixes including procedure handling, DDL panic prevention, and task list display issues, with links to documentation and related articles.
This article explains Elasticsearch's cluster architecture, including nodes, indices, shards, and replicas, compares mixed and tiered deployment models, discusses the data storage layer and replication trade‑offs, and presents two typical distributed data system designs with their advantages and drawbacks.
This article explains what geo‑active multi‑active (异地多活) architecture is, why it is needed for ultra‑high availability, and walks through the step‑by‑step evolution from a single‑node system to sophisticated multi‑data‑center designs that use redundancy, disaster‑recovery, data synchronization, routing, and conflict‑resolution techniques.
Covering essential high‑availability techniques, the article examines disaster‑recovery architectures from same‑city dual‑center to cross‑country active‑passive deployments, compares five patterns, details three multi‑active models, outlines required traffic‑scheduling, replication, and database layers, and provides design methodology, practical safeguards, and key HA metrics.
This article provides a comprehensive overview of disaster recovery, covering its definition, the distinction between backup and DR, their respective roles, key metrics such as RPO and RTO, various replication technologies, and practical implementation methods across storage, network, and host layers.
This article explains the principles, evolution, and implementation details of high‑availability architectures—from basic single‑node setups to multi‑active, cross‑region deployments—covering redundancy, disaster recovery, data synchronization, routing strategies, and the challenges of achieving true geo‑distributed active‑active systems.
This article provides a comprehensive overview of HDFS, covering basic file system concepts, HDFS architecture, high availability, federation, replica placement, storage policies, colocation, data integrity, and key design considerations for large‑scale distributed storage.
This article explains the concept of anti‑entropy as a key mechanism for achieving eventual consistency in distributed database systems, illustrates how XDB Enterprise handles node failures and data drift, and shows practical examples of repairing inconsistencies using the AE service.
This article traces the evolution of database systems—from the first network and hierarchical models through relational databases, NoSQL sharding, and finally modern distributed SQL—explaining why distributed databases emerged, how they handle data distribution, consistency, SQL-to‑KV mapping, and transaction challenges.
This article examines how enabling the MTS (writeset‑based incremental parallel replay) feature in DTLE and MySQL affects data transfer speed, CPU and memory usage, and network latency, presenting test procedures, configurations, and performance results that show up to 28‑fold speed improvements.
This article explains how 360 Commercialization implements cross‑IDC hot‑standby for Kafka using MirrorMaker 2, covering MM2 fundamentals, architecture, internal topics, deployment on Kubernetes, design goals, solution details, challenges such as dynamic configuration and offset reverse‑mapping, and productized risk mitigation.
This article explores the core concepts, design requirements, architectural models, scalability, high availability, performance optimization, and security considerations of distributed file systems, comparing centralized and decentralized approaches while highlighting practical solutions for persistence, consistency, and fault tolerance.
This article introduces four change‑data‑capture solutions—Canal, Maxwell, Databus, and Alibaba Data Transmission Service (DTS)—explaining their principles, processing steps, features, and practical advantages for real‑time data synchronization and migration in big‑data environments.
The article uses a whimsical story of a coronavirus‑like virus transmitted from a bat to humans to illustrate the Gossip protocol, its three functions—direct mail, anti‑entropy, and epidemic spread—and discusses their advantages, drawbacks, and practical applications in achieving eventual consistency in distributed systems.
The guide explains how to configure Tencent Cloud Elasticsearch clusters for multi‑AZ high availability by using zone‑aware node attributes, deploying data nodes in multiples of AZs, assigning three dedicated masters across zones, setting replica shards and force‑awareness rules, and safely upgrading single‑AZ clusters without service interruption.
This article explains how Ele.me built a multi‑active, geographically distributed system that enables elastic scaling and data‑center‑level disaster recovery by partitioning services, routing traffic, replicating data in real time, and enforcing strict consistency and availability principles.
This article breaks down the five major data replication approaches—application‑level, host‑level, database‑level, storage‑gateway, and storage‑media—detailing their principles, advantages, drawbacks, and typical use cases to help professionals design effective disaster‑recovery solutions.
The article introduces the 2021 China Disaster Recovery Whitepaper and explains five layers of data replication—application, host, database, storage‑gateway, and storage‑media—detailing their mechanisms, advantages, limitations, and use cases in modern backup and business continuity solutions.
This article outlines why distributed systems are needed, explains how they address cost and high‑availability issues by coordinating cheap nodes, and discusses the new coordination challenges such as service discovery, load balancing, fault isolation, monitoring, data partitioning, replication, and distributed transactions, providing a roadmap for further study.
An in‑depth look at how JD Cloud’s high‑reliability storage architecture tackles data reliability challenges—covering replica management, redundancy, detection and repair mechanisms, tiered storage designs, and Intel Optane’s role in boosting performance—offering practical strategies for balancing cost and resilience.
This article outlines why distributed systems are needed, explains how they address cost and high‑availability issues through coordinated nodes, and discusses the new challenges such as service discovery, load balancing, avalanche prevention, monitoring, data sharding, replication, and distributed transactions, while offering practical and theoretical learning paths.
This article provides a detailed collection of SQL Server commands and techniques covering database creation, table and index management, common DML statements, advanced set operators, various join types, grouping, pagination, maintenance tasks, replication setup, linked‑server usage, and a synchronization stored procedure.
This article provides a comprehensive technical overview of Tencent Cloud's TBase database, covering fundamental database classifications, TBase's cloud‑native architecture, distributed transaction mechanisms, storage and compute separation, security layers, backup strategies, and real‑world deployment cases.
The article describes Ctrip's Data Replicate Center (DRC), a MySQL middleware that enables real‑time bidirectional replication across data‑center clusters, detailing its architecture, low‑latency optimizations, consistency mechanisms, DDL handling, monitoring, and future high‑availability improvements.
This article details a real‑world implementation of Oracle GoldenGate for multi‑source data distribution, covering the architectural design, OGG fundamentals, operational challenges, troubleshooting steps, and future upgrade directions in a production environment.
This article introduces Alibaba's Canal project, explaining its background, architecture, and how it parses MySQL binlog for real‑time data subscription, and provides a complete Java client example with code snippets to demonstrate connecting, fetching, and processing binlog events.
Canal, an open‑source Alibaba project, mimics MySQL slave behavior to subscribe to binlog events, parses them, and supports both standalone and ZooKeeper‑coordinated cluster deployments, offering flexible state storage, message processing pipelines, and integration options such as TCP, Kafka, and RocketMQ for real‑time data synchronization.
This article provides a comprehensive overview of HDFS, covering its purpose, architecture, read/write processes, component roles, command-line tools, replica placement strategies, and the advantages and disadvantages of using Hadoop's distributed file system for large-scale data storage.
This article introduces the global data replication market, explains synchronous and asynchronous replication, and details five typical replication techniques—host‑based, application/middleware‑based, database‑based, storage‑gateway‑based, and storage‑media‑based—highlighting their principles, advantages, and trade‑offs for disaster‑recovery planning.
This article explains distributed data consistency concepts, the CAP theorem, various consistency levels, and provides detailed overviews of three major consensus algorithms—Paxos, ZAB, and Raft—including their mechanisms, roles, and practical applications such as in CB‑SQL.
This article explains the storage challenges of big data, introduces RAID technologies and their variants, and shows how the principles of RAID are applied in the Hadoop Distributed File System (HDFS) to achieve scalable, reliable, and high‑performance data storage and processing.
This article traces the evolution of Alibaba’s distributed data layer—from the early Cobar system to the modern TDDL framework and DRDS service—explaining their architectures, limitations, sharding principles, transaction‑boundary strategies, heterogeneous index tables, and the Jingwei data‑replication platform that together enable seamless scaling and high‑performance SQL processing across thousands of databases.
The article presents DTLE, an open‑source MySQL data‑transfer middleware that extends replication capabilities with high‑performance CDC, multi‑topology support, cloud‑to‑cloud synchronization, and robust cluster management, while comparing it with other open‑source solutions and showcasing real‑world demos.
This talk details Ele.me’s three‑month rollout of a dual‑active (multi‑active) database system, covering design principles, architecture, migration steps, challenges such as data consistency and DDL handling, the tools built (DRC, D‑Bus, DCP, EMHA), and the performance and reliability benefits achieved.
The article examines the advantages and disadvantages of microservice architectures, discussing flexibility, scalability, autonomy, monitoring, build and release complexity, security, and data replication, while highlighting practical trade‑offs and lessons learned from real‑world implementations.
The article details Veritas Velocity’s hybrid‑cloud data replica solution, its architecture, deployment steps, role‑based access, workflow automation with Oracle RMAN, and performance benchmarks that show up to 8.8× faster test‑environment provisioning while halving required manpower.
This article details how Ele.me engineered a cross‑region active‑active system that scales elastically, tolerates whole‑data‑center failures, and maintains real‑time food‑delivery performance through geographic sharding, intelligent routing, and robust data‑replication middleware.
This article explores the importance of data consistency in distributed systems, illustrating real‑world scenarios, explaining consistency models such as strong, weak and eventual, and detailing the challenges and theories like CAP and BASE that guide system designers in balancing consistency, availability, and partition tolerance.
Kafka MirrorMaker provides near‑real‑time cross‑data‑center replication by consuming from a source cluster and producing to a target cluster, and this guide explains its core features, new vs. old consumer APIs, partition assignment strategies, performance tuning, network considerations, and practical command‑line examples.
The article explains NetApp MetroCluster’s clustered storage architecture, including dual‑site HA pairs, 4‑node active‑active designs, synchronization mechanisms such as SyncMirror, ClusterRemote and CRS, and the network and NVRAM strategies that enable seamless data protection and disaster recovery across distances up to 200 km.
This article details Ele.me's design and implementation of a MySQL bidirectional replication component (DRC) that enables sub‑second, high‑throughput data synchronization across Beijing and Shanghai data centers, addressing latency, consistency, and failover challenges in a multi‑active environment.
This note distills Jay Kreps' extensive blog on logs, explaining their core role in distributed databases, real‑time data pipelines, replication, and state‑machine consistency, and showing how logs unify concepts from version control to streaming architectures.
This article traces the history of disaster recovery from its 1970s origins through the rise of hot‑standby sites, the impact of internet expansion, and the shift to cloud‑based DRaaS, while forecasting future trends such as integrated business‑as‑disaster solutions, automated management platforms, and social‑media‑driven emergency communication.
This article explains how HDFS, inspired by Google’s GFS, provides a low‑cost, highly reliable, fault‑tolerant, and high‑performance distributed file system for big‑data workloads by using replication, standby NameNodes, block storage, rack awareness, and compute‑close‑to‑data strategies.
This article provides a comprehensive overview of Oracle GoldenGate, detailing its supported databases, modular architecture, key components such as Extract, Data Pump, Replicat, Trails, Checkpoints, Manager and Collector, as well as processing types, group configuration, and commit sequence numbers for reliable data replication.
This guide explains how to restore Oracle GoldenGate synchronization after a source‑side crash by repairing the extract service, restarting the pump process, and correcting the replicat on the target, with detailed commands and file‑selection procedures.
This article provides a comprehensive overview of Hadoop Distributed File System (HDFS), detailing its design goals, architecture components such as NameNode, DataNode and SecondaryNameNode, data block handling, replication strategies, communication protocols, and the read, write, and delete processes.
This article explains how the Tencent Game DBA team uses Percona's pt-table-sync to detect and automatically repair MySQL replication inconsistencies, achieving up to 30‑fold speed improvements, reducing resource usage, and enabling a data self‑healing service for large‑scale gaming databases.
This article explains why DB2 customers must upgrade to avoid EOS risks, compares replication options, and provides a detailed CDC‑based step‑by‑step process to achieve minute‑level downtime, zero performance impact, and fast rollback for large‑scale DB2 version migrations.
This article explains how HDFS abstracts multiple servers into a single file system, splits files into replicated blocks, manages metadata via NameNode and DataNode, and provides linear capacity scaling and high reliability for big data workloads.
This article chronicles the development, design decisions, and technical achievements of Alibaba's Data Replication Center (DRC), a real‑time heterogeneous database synchronization platform that has become a core infrastructure for multi‑active data centers, large‑scale e‑commerce, and cloud services.
The article explains why consistency is essential in distributed systems, describes the CAP theorem, outlines various consistency models such as strong, weak, and eventual consistency, and discusses the trade‑offs between data correctness and system performance.
The article explains how JD.com designed and deployed a multi‑center transaction architecture, using a high‑performance data bus and strict consistency and routing controls, to handle the massive traffic spikes of the 11.11 e‑commerce event while ensuring scalability and disaster recovery.
The article presents an overview of Alibaba's Data Replication Center (DRC), detailing its architecture, real‑time cross‑region synchronization capabilities, consistency and latency guarantees, deployment strategies, and its use cases on Alibaba Cloud such as RDS migration and multi‑active e‑commerce workloads.
The 2015 Beijing QCon highlighted four technical talks covering asynchronous processing in distributed systems, the DRC data‑replication infrastructure, minute‑level high‑availability fault recovery, and cloud‑era database operations, illustrating Alibaba's approaches to scalability and reliability in modern cloud platforms.
