This article examines the challenges of high‑concurrency flash‑sale scenarios in multi‑active architectures, analyzes fairness issues caused by geographic latency, and presents practical solutions such as data‑sharding and global‑clock ordered queues, complemented by a Redis‑based implementation example.
This article analyzes the "write‑after‑immediate‑read" challenge in multi‑active disaster‑recovery setups, breaks down solution directions, presents a three‑city five‑center case study, and outlines a four‑step model—distinguish scenarios, mark written data, assess latency, and enable near‑by access—to achieve consistent, low‑latency reads.
This article examines the challenges of building multi‑active distributed systems, focusing on the data layer’s role in high availability, write‑latency, sharding, isolation, replication strategies, and routing decisions, and provides concrete architectural patterns and practical guidelines for robust backend design.
The article examines the write‑after‑read consistency problem in multi‑active cross‑region systems, compares single‑write‑single‑read routing, quorum‑based multi‑write‑multi‑read, and read‑while‑copy methods, explains why primary‑secondary replication is preferred, and proposes a four‑step framework—scenario flagging, data marking, latency evaluation, and near‑by asynchronous replication—to meet WAR requirements efficiently.
This article analyzes the architectural challenges of building large‑scale distributed systems with multi‑active (cross‑city) capabilities, focusing on data‑layer design, write latency, replication models, sharding techniques, and routing impacts to guide reliable, high‑performance infrastructure decisions.
This article analyzes how write latency, write volume, isolation, and data replication strategies influence the design of multi‑active distributed systems, offering practical guidance on sharding, synchronous and asynchronous replication, routing, and architecture selection for high availability and performance across regions.
This article explains Bilibili's technical strategies—preloading, circuit breaking, sharding, multi‑active deployment, and Kubernetes auto‑scaling—that ensure the live‑gift panel, feeding flow, and supporting services maintain 99.99% uptime even during massive traffic spikes.
This article explains the motivations, application scenarios, architectural patterns (same‑city, cross‑city, and cross‑country), and concrete design techniques for building multi‑active cross‑region systems that ensure high availability and graceful degradation during extreme failures.
To achieve disaster-recovery and cross-data-center resilience, Didi progressed from a simple proxy double-write scheme to a sophisticated MQ-free multi-active Redis design that uses a dedicated syncer, shard-based loop prevention, op-id replay protection, conflict detection, and incremental AOF durability, ensuring low latency, no data loss, and consistent availability.
Bilibili guarantees 99.99% availability for its live‑gift ecosystem by pre‑loading panel data, using circuit‑breakers and cached responses, sharding databases, auto‑scaling Kubernetes pods, employing Redis hot‑key caching and distributed rate limiting, handling order timeouts and idempotent MQ processing, and deploying multi‑active cross‑region services with intelligent discovery.
The article examines Vipshop's massive March 2023 outage caused by an IDC cooling failure, critiques superficial PPT‑driven reliability claims, and offers practical SRE insights on fault drills, true multi‑active architectures, and how ops teams can gain influence despite budget constraints.
This article details Bilibili's SRE team's design and implementation of a high‑availability multi‑active architecture, covering zone types, same‑city and cross‑region deployments, traffic routing, cache consistency, message handling, governance, and practical lessons learned from real‑world incidents.
The article explains various high‑availability solutions for stateful backend services, comparing cold backup, dual‑machine active/standby, same‑city active‑active, cross‑city active‑active, and cross‑city multi‑active approaches, and discusses their trade‑offs, implementation details, and real‑world examples from large internet companies.
This article presents a comprehensive guide to microservice architecture evolution, covering routing, multi‑active deployment, gray releases, canary/rolling/blue‑green strategies, traffic shading, rate limiting, and practical implementation steps using cloud‑native gateways, service meshes, and Kubernetes labeling.
This article chronicles Bilibili's five‑year evolution of Site Reliability Engineering, detailing the introduction of SRE culture, the construction of high‑availability and multi‑active architectures, capacity management with Kubernetes, VPA/HPA, incident case studies, and the ongoing transformation of SRE practices across the organization.
This guide outlines the motivations, architectural guidelines, routing strategies, RPC and message replication techniques, storage synchronization methods, and traffic‑switching procedures needed to successfully implement a multi‑active, cross‑region system.
The article visually details Bilibili’s multi‑active database architecture, showing system layout, data replication, failover strategies, and operational workflows that enable high availability across multiple data centers by synchronizing data, balancing load, and maintaining continuous service during node or regional failures.
This article examines high‑availability strategies for stateful backend services, comparing cold backup, hot standby, same‑city active‑active, cross‑region active‑active, and multi‑active architectures, highlighting their advantages, limitations, and practical implementation considerations such as downtime, data loss, synchronization overhead, and conflict resolution.
This article explores the evolution of high‑availability strategies for stateful backend services, comparing cold backups, active/standby, same‑city and cross‑city active‑active setups, and discusses the trade‑offs, design considerations, and real‑world implementations of multi‑active and multi‑active architectures.
This article explains the design and implementation steps for a multi‑active (geo‑distributed) system, covering data‑center segmentation, traffic scheduling, RPC routing, database and Redis sharding, distributed IDs, message queue handling, job coordination, and cut‑over procedures.
This post‑mortem details the July 2021 Bilibili outage caused by a Lua bug in the OpenResty‑based SLB, describing the timeline, root‑cause analysis, mitigation steps, and the technical and organizational improvements implemented to prevent similar incidents.
This article explains TiDB's multi‑active high‑availability architectures—including same‑city dual‑center, triple‑center, and two‑region three‑center deployments—detailing hard requirements, RPO/RTO goals, placement‑rule configurations, and practical disaster‑recovery recommendations for distributed database clusters, and how adaptive sync modes affect failover performance.
This article explains what geo‑active multi‑active (异地多活) architecture is, why it is needed for ultra‑high availability, and walks through the step‑by‑step evolution from a single‑node system to sophisticated multi‑data‑center designs that use redundancy, disaster‑recovery, data synchronization, routing, and conflict‑resolution techniques.
Despite widespread cloud adoption, large‑scale outages still occur, prompting Alibaba Cloud’s high‑availability team to share the evolution, principles, and open‑source implementation of multi‑active disaster recovery (AppActive) that aims to achieve minute‑level failover and near‑zero downtime.
Covering essential high‑availability techniques, the article examines disaster‑recovery architectures from same‑city dual‑center to cross‑country active‑passive deployments, compares five patterns, details three multi‑active models, outlines required traffic‑scheduling, replication, and database layers, and provides design methodology, practical safeguards, and key HA metrics.
This article presents a step‑by‑step guide for constructing a hybrid cloud disaster‑recovery solution using MSHA's multi‑active architecture, covering business background, design challenges, dual‑active deployment, traffic routing, data synchronization, one‑click failover, and validation of sub‑minute RPO/RTO for e‑commerce platforms.
The article explains why large internet firms adopt cross‑region multi‑active architectures for high availability, compares cold backup, hot standby, same‑city active‑active, and cross‑region active‑active solutions, discusses their trade‑offs, and presents practical design patterns and questions for implementing such systems.
This article examines stateful backend services and compares various high‑availability strategies—including cold backup, dual‑machine hot standby, same‑city and cross‑city active‑active, and multi‑active architectures—highlighting their benefits, drawbacks, and practical implementation considerations.
This article explains the evolution of high‑availability strategies for stateful backend services, comparing cold backup, dual‑machine hot standby, same‑city active‑active, cross‑city active‑active and multi‑active solutions, and discusses their trade‑offs, implementation details, and practical considerations.
This article examines high‑availability strategies for stateful backend services, covering cold backup, dual‑machine hot standby, same‑city active‑active, and remote multi‑active solutions, while discussing their benefits, trade‑offs, and architectural patterns for resilient distributed systems.
This article examines the evolution of high‑availability architectures—from cold backup to multi‑active deployments—detailing the technical trade‑offs, design patterns, and real‑world implementations used by large internet companies such as Alibaba, Eleme, and Taobao.
This article explains how backend services can be classified as stateless or stateful and explores a range of high‑availability strategies—from simple cold backups and active‑standby setups to same‑city, cross‑city, and multi‑active architectures—highlighting their trade‑offs and implementation considerations.
This article analyzes the evolution of high‑availability deployment—from cold backup to cross‑region multi‑active—explaining the trade‑offs of each solution, the challenges of stateful services, and real‑world architectures used by companies like Alibaba and Eleme.
This article examines stateful backend services and various high‑availability strategies—including cold backup, active/standby hot backup, same‑city active‑active, cross‑region active‑active, and multi‑active architectures—detailing their advantages, limitations, and practical implementation considerations, and includes real‑world examples from major e‑commerce platforms.
This article explains the evolution of high‑availability deployment architectures—from simple cold backups and hot standby to same‑city active‑active, cross‑city active‑active, and finally multi‑active solutions—detailing their advantages, drawbacks, and practical design considerations for large‑scale internet services.
This article explains how Ele.me built a multi‑active, geographically distributed system that enables elastic scaling and data‑center‑level disaster recovery by partitioning services, routing traffic, replicating data in real time, and enforcing strict consistency and availability principles.
Multi‑active high‑availability architectures—ranging from same‑city dual‑active and two‑site three‑center setups to fully remote multi‑active deployments—provide continuous 24/7 service by replicating data across sites, but introduce latency, consistency, routing, and cost complexities that require careful unit‑based design, synchronized storage, and sophisticated traffic management.
This article describes JDHBase’s large‑scale KV storage, its HBase‑based replication principle, the multi‑active cluster architecture with Fox Manager, client routing, automatic failover, dynamic replication tuning, serial replication guarantees, and future directions for improving cross‑region disaster recovery.
This article describes JDHBase’s large‑scale KV storage architecture, its HBase‑based asynchronous replication mechanism, multi‑active cluster design, client‑side routing via Fox Manager, automatic failover strategies, dynamic replication tuning, and serial replication techniques to ensure data consistency across geographically distributed data centers.
JDHBase, JD.com’s large‑scale KV store, powers billions of daily reads and writes across 7,000 nodes, and this article details its multi‑active, cross‑region architecture—including HBase replication fundamentals, Fox Manager routing, automatic failover policies, dynamic replication tuning, and serial replication to ensure strong consistency.
This article describes JDHBase’s large‑scale KV storage deployment, its HBase‑based asynchronous replication mechanism, the multi‑active architecture with active‑standby clusters, client interaction via Fox Manager, automatic failover strategies, dynamic replication tuning, and serial replication techniques to ensure data consistency across data centers.
Wang Xiaobo explains how public‑cloud services can simplify designing and implementing active‑active architectures, covering data‑center redundancy, real‑time synchronization, fault‑tolerant networking, micro‑service migration, and cost‑benefit trade‑offs, while urging incremental, cloud‑assisted approaches rather than full 100% multi‑active deployments.
This article explains the concepts of unitization and multi‑active data centers, analyzes the drawbacks of single‑IDC deployments, and provides a detailed guide on implementing MySQL binlog synchronization while addressing common challenges such as data loops, duplicate inserts, unique‑index conflicts, and DDL handling, including practical solutions like GTID and auxiliary tables.
This article explains DNS fundamentals, the step‑by‑step resolution process, TTL caching, and how DNS‑based Global Server Load Balancing (GSLB) can direct traffic to the nearest active data‑center, providing a practical guide for building multi‑active, high‑availability infrastructures.
This talk details Ele.me’s three‑month rollout of a dual‑active (multi‑active) database system, covering design principles, architecture, migration steps, challenges such as data consistency and DDL handling, the tools built (DRC, D‑Bus, DCP, EMHA), and the performance and reliability benefits achieved.
The article describes how a DBA team at Ele.me transformed manual database management into a fully automated, platform‑driven operation covering monitoring, alarm handling, MHA automation, resource pooling, large‑scale migration, SQL review, and multi‑active DDL release using Go‑based tools and custom workflows.
This article describes how the Beetle efficiency platform addresses multi‑active deployment challenges by solving session‑sharing, distributed locking, and task‑recovery problems to enable seamless service upgrades and bug fixes without user disruption.
The talk details Ele.me’s rapid growth from 300k to over 10 million daily orders, describing the challenges of high‑concurrency, multi‑active micro‑service architecture, IDC planning, database refactoring, disaster‑recovery, NOC operations, and the systematic processes that enabled stable, scalable delivery across two data centers.
This article describes how Alibaba's NineGame access system moved from traditional system‑centric high‑availability designs to a business‑oriented, three‑dimensional architecture that enables 3‑minute problem detection, 5‑minute service restoration, and an average of one outage every two months.
This article details JD.com's multi-active distributed architecture, covering its evolution from single‑data‑center to multi‑region deployments, network design, leaf‑spine topology, data consistency mechanisms, application scheduling, monitoring, and disaster recovery strategies that enhance high availability and user experience.
The article details Suning’s decade‑long evolution of its membership platform for the Double‑11 shopping festival, covering early offline CS architecture, the transition to a WebSphere‑based e‑commerce system, vertical module splitting, data migration with Spark and Hive, and the implementation of same‑city multi‑active deployment.
This article details Ele.me's design and implementation of a MySQL bidirectional replication component (DRC) that enables sub‑second, high‑throughput data synchronization across Beijing and Shanghai data centers, addressing latency, consistency, and failover challenges in a multi‑active environment.
The article describes how Alibaba's game integration platform achieved business‑oriented high availability by abandoning traditional system‑centric designs and implementing a three‑dimensional architecture that combines clear HA goals, multi‑active deployment, client‑side retries, functional isolation, automated monitoring, and rapid fault recovery, ultimately meeting a 3‑minute issue‑location and 5‑minute business‑recovery target.
The article details Alibaba's three‑stage evolution from same‑city active‑active to two‑site three‑center disaster recovery and finally to geo‑distributed multi‑active architecture, explaining its motivations, technical challenges such as latency and data consistency, the solutions implemented, and the resulting scalability and fault‑tolerance benefits.
The article details Weibo's evolution toward multi‑active cross‑datacenter deployment, describing initial motivations, early synchronization attempts, the eventual MCQ‑based messaging solution, operational challenges such as latency and data consistency, and best‑practice recommendations for large‑scale distributed systems.
In this interview, Alibaba’s Lin Hao (aka Bi Xuan) explains the motivations, deployment details, technical challenges such as latency, routing consistency, and data consistency, and the solutions behind the company’s multi‑active disaster‑recovery data‑center architecture that powers its massive e‑commerce platform.
This interview with Alibaba's Lin Hao (aka Bi Xuan) explains why the company pursued a multi‑active, geographically distributed data‑center architecture, how it was deployed for the 2014 Double‑11 event, and the technical challenges—including latency, routing consistency, and data synchronization—that had to be solved to achieve high availability and low‑cost disaster recovery.
