This article explores the advantages of unitized architecture over traditional microservices, detailing how its modular design, dedicated routing layer, and tailored observability practices enhance system resilience, fault‑tolerance, and operational insight for large‑scale distributed applications.
This article analyzes Baidu's Baige 4.0 AI infrastructure, detailing its four‑layer architecture, XMAN 5.0 hardware, HPN network, BCCL communication library, and AIAK inference upgrades, and explains how these innovations address large‑model training and inference challenges while boosting performance, utilization, and cost efficiency.
AI infrastructureGPU accelerationHigh-Performance Computing
This article explains Redis split‑brain behavior, describing its definition, causes such as network failures and Sentinel elections, the resulting data loss during master‑slave switches, and practical prevention measures including quorum configuration, timeout tuning, network monitoring, proxy layers, and the min‑slaves‑to‑write and min‑slaves‑max‑lag settings.
This paper introduces the concept of software antifragility, drawing on Taleb’s theory to argue that embracing errors through fault tolerance, automatic runtime repair, and fault injection can transform software systems into self‑improving, more robust entities, and discusses implications for development processes and product reliability.
The article explains how interface‑level faults—where the system stays up but business performance degrades—can be mitigated through four core techniques (degradation, circuit breaking, rate limiting, and queuing), detailing their principles, implementation patterns, and practical trade‑offs for backend services.
Kubernetes aligns perfectly with AI inference demands by offering built‑in scalability, resource and performance optimization, seamless portability across clouds, and robust fault‑tolerance, making it a cost‑effective, high‑availability foundation for deploying large‑scale machine‑learning models.
This article explains how to keep MySQL and Elasticsearch data in sync using synchronous calls, asynchronous notifications, or binlog listeners, and dives deep into Elasticsearch cluster design, node roles, distributed storage, query phases, split‑brain handling, and fault‑tolerance mechanisms.
This article translates and analyzes the MegaScale system—co‑developed by ByteDance and Peking University—that enables efficient, stable training of massive language models on clusters of more than 10,000 GPUs, achieving 55.2% MFU and a 1.34× speedup over Megatron‑LM.
The article explains how to build reliable microservices by isolating failures, applying graceful degradation, change‑management, health checks, self‑healing, fallback caching, retry strategies, rate limiting, fast‑fail principles, circuit breakers, and failure‑testing to ensure high availability in distributed cloud‑native systems.
This article explains ten core techniques—system splitting, decoupling, asynchrony, retry, compensation, backup, multi‑active strategies, isolation, rate limiting, circuit breaking, and degradation—that together enable robust, high‑availability architectures for modern backend services.
High Availabilitydistributed systemsfault tolerance
The article explains Ctrip's service registration center architecture, including its two‑layer Data and Session design, multi‑sharding, fault‑tolerance mechanisms, Redis‑based cluster discovery, design trade‑offs such as proxy versus Smart SDK, hashing strategy, and operational considerations for burst traffic and future scaling.
This article explains the Paxos consensus algorithm, detailing its roles, three-phase execution process, key properties such as consistency, availability and fault tolerance, and showcases its practical applications in distributed databases, file systems, and coordination services.
This article outlines six fundamental microservice splitting principles—including single responsibility, appropriate granularity, interface segregation, product impact avoidance, scalability, and fault tolerance—to help architects design maintainable, decoupled, and resilient services.
Single Responsibilityfault toleranceinterface segregation
DeepRec Extension enhances large‑scale sparse model training by adding automatic elastic training, resource‑aware scheduling, real‑time monitoring, and efficient fault‑tolerance mechanisms, enabling lower cost, higher throughput, and more reliable distributed training for AI workloads.
This article provides a comprehensive overview of Apache RocketMQ, detailing its core components, producer and consumer workflows, storage strategies, master‑slave synchronization, Raft‑based half‑write and leader election mechanisms, and best‑practice recommendations for high‑throughput, fault‑tolerant messaging systems.
This article explains how DataMesh, a sidecar‑deployed Redis proxy, uses a layered architecture, risk analysis, sub‑second recovery mechanisms, large‑scale deployment strategies, and fault‑transfer capabilities to consistently meet a five‑nine service level agreement.
The article examines how a payment platform improves its 24/7 availability by preventing failures, detecting incidents in real time, and implementing rapid recovery measures such as dynamic routing, resource limits, monitoring, logging, and service degradation, while sharing practical Q&A insights.
The article analyzes ByteDance and Peking University's MegaScale system that enables efficient, stable training of large language models on clusters exceeding ten thousand GPUs, detailing algorithmic tweaks, 3D parallel communication overlap, operator optimizations, data‑pipeline improvements, network tuning, and fault‑tolerance mechanisms that together achieve a 55.2% MFU on a 175B model.
The article walks through nine practical techniques—system splitting, decoupling, asynchronous processing, retry, compensation, backup, multi‑active deployment, rate limiting, circuit breaking, and degradation—explaining why each is needed, how they are implemented in real‑world microservice architectures, and what trade‑offs to consider.
Circuit BreakerHigh Availabilitydistributed systems
This article examines how database middleware tackles the demanding needs of large‑scale internet services by providing centralized connection‑pool management, transparent read‑write splitting, diverse load‑balancing algorithms, sharding support, automatic failover, security controls, comprehensive monitoring, and flexible backup‑recovery mechanisms.
To ensure the massive, multi‑platform membership service remains fast and reliable, this article details a multi‑center architecture using Elasticsearch for unified member data, Redis caching, and MySQL partitioning, along with traffic isolation, fault‑tolerant syncing, and fine‑grained flow‑control and degradation strategies.
This article compares RabbitMQ and Apache Kafka across architecture, message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, and provides guidance on which platform suits specific use‑cases such as flexible routing, strict ordering, long‑term retention, or high throughput.
This article explains what Redis hot keys are, the scenarios that generate them, their risks, and provides practical monitoring methods and mitigation strategies—including cache pre‑warming, distributed caching, rate limiting, and secondary caches—to keep production systems stable.
The article analyzes the November 27 Didi outage caused by an aggressive Kubernetes upgrade, then presents four engineering principles—controlling cluster size, eliminating single points of failure, treating restarts as normal, and decoupling data and control planes—to build more resilient cloud‑native systems.
This article introduces Resilience4j, a lightweight fault‑tolerance library for Spring Boot, explaining its core decorators—CircuitBreaker, TimeLimiter, Bulkhead, and RateLimiter—along with configuration examples, annotation usage, fallback handling, and practical test code to improve system stability and resilience.
This article outlines ten practical techniques—including system splitting, decoupling, asynchronous processing, retry strategies, compensation, backup, multi‑active deployment, isolation, rate limiting, circuit breaking, and degradation—to help engineers design highly available, resilient architectures for large‑scale internet applications.
This article provides a comprehensive overview of RAID technology, explaining disk groups, virtual disks, detailed characteristics of RAID 0, 1, 1ADM, 5, 6, 10, 10ADM, 1E, 50, and 60, and compares their fault tolerance, I/O performance, and storage capacity considerations.
This article explains RAID fault‑tolerance mechanisms, consistency verification, hot‑spare and emergency backup, rebuild processes, virtual‑disk read/write policies, power‑loss protection, disk striping, mirroring, foreign configurations, power‑saving and pass‑through features, providing a comprehensive overview of modern storage system capabilities.
The article explains the stability challenges of modern microservice architectures, outlines the three governance domains (development/testing, change, runtime), and introduces OpenSergo’s open, cloud‑native specifications, control‑plane, and data‑plane solutions for traffic routing, gray‑release, and fault‑tolerance.
This article breaks down high‑availability system design into six critical layers—architecture, development standards, application services, storage, product safeguards, and operations—offering concrete practices such as capacity planning, fault‑tolerant patterns, monitoring, and incident‑response strategies to achieve four‑nine (99.99%) uptime.
This guide explains essential Kafka monitoring techniques, third‑party tools, custom scripts, key metrics, and practical strategies for high availability, fault detection, rapid recovery, and ongoing testing to keep Kafka clusters stable and performant.
To improve stability and reduce costs during high‑traffic events, we replaced the Zookeeper‑dependent TBSchedule framework with a lightweight, Redis‑based distributed scheduler that decentralizes task execution, uses thread pools instead of timers, and supports dynamic scaling and seamless degradation for reliable order processing.
This article explains why circuit breakers are essential in microservice architectures, introduces Netflix's Hystrix library, details its design principles, shows step‑by‑step demos for Ribbon and Feign integration, and covers dashboards, Turbine, isolation strategies, request merging, caching, and related Spring Boot SPI mechanisms.
The article systematically defines software architecture, distinguishes systems, subsystems, modules, and components, compares frameworks with architectures, explores TOGAF and RUP classifications, traces the evolution from monoliths to micro‑services, and presents concrete design principles and common pitfalls for building scalable, maintainable systems.
This article explains essential fault‑tolerance patterns for microservices—including timeouts, retries, circuit breakers, distributed deadlines, and rate limiting—detailing their basic forms, drawbacks, and practical implementation strategies to improve reliability and prevent cascading failures.
The article examines Vipshop's massive March 2023 outage caused by an IDC cooling failure, critiques superficial PPT‑driven reliability claims, and offers practical SRE insights on fault drills, true multi‑active architectures, and how ops teams can gain influence despite budget constraints.
The article explores how AI's rise highlights human error, argues that fault‑tolerant design is essential for user‑centric products, outlines practical guidelines such as anticipatory guidance and constructive error recovery, and validates the approach with a VR‑tool experiment.
This article describes the design, technology choices, architecture, and implementation details of an automated payment channel management system that uses Redis‑based time‑series storage, custom circuit‑breaker logic, and monitoring to achieve fast fault detection, accurate alerting, and future automated failover.
This article explains the Raft consensus algorithm, detailing its roles, leader election process, term management, fault handling, and how it ensures consistency in both single‑node and multi‑node distributed systems for modern cloud‑native applications.
The article explains how system failures like bugs, security breaches, and cloud outages can cripple businesses, and outlines the concepts of fault tolerance and disaster recovery as essential components of high‑availability architectures to ensure continuous service and protect revenue.
The article outlines a comprehensive high‑availability architecture framework across six layers—development standards, application services, storage, product fallback, operations deployment, and emergency response—detailing design principles such as stateless services, elastic scaling, redundant storage, robust monitoring, gray releases, and chaos engineering to ensure resilient, continuously available systems.
The article describes the design and implementation of Taobao Dou Dizhu’s new single‑player endgame mode, which generates daily unique puzzles with a guaranteed single solution, manages activity triggers, Redis caching, AI interaction, fault tolerance, consistency, and reward idempotency, boosting user retention during promotions.
This article explains how QQ Music builds a high‑availability system by combining redundant architecture, a comprehensive toolchain—including chaos engineering and full‑link pressure testing—and deep observability to gracefully handle failures in a large‑scale microservice environment.
This article explains why high availability is crucial for backend services and outlines ten practical design principles—including dependency control, avoiding single points, load balancing, isolation, rate limiting, circuit breaking, async processing, degradation, gray release, and chaos engineering—to help developers create resilient APIs.
The article explains why high availability is essential for backend services, defines its core concepts, and outlines key design principles such as minimizing dependencies, avoiding single points of failure, load balancing, resource isolation, rate limiting, circuit breaking, asynchronous processing, degradation strategies, gray releases, and chaos engineering to build resilient APIs.
This article breaks down the essential design and operational considerations for achieving high availability across six layers—development standards, application services, storage, product strategy, operations deployment, and incident response—providing concrete practices, metrics, and safeguards to reach four‑nine (99.99%) uptime.
This article presents a step‑by‑step experiment on a Cassandra cluster spanning two data centers, demonstrating how token ownership, data distribution, and fault‑tolerance behave when nodes fail or are removed, and explains the observed owns percentages and replication effects.
QQ Music achieves high availability by layering redundant multi‑datacenter architecture, proactive chaos‑engineering toolchains, and comprehensive observability—including metrics, logging, tracing and profiling—while employing service grading, adaptive retry windows and EMA‑based dynamic timeouts to gracefully handle faults across its massive micro‑service ecosystem.
High Availabilitychaos engineeringdistributed systems
This article systematically explores high‑availability system design from development standards, capacity planning, application services, storage, product strategies, operations deployment, to incident response, presenting key concepts, architectural patterns, and practical guidelines for building resilient services.
This article explains how to design and operate resilient microservice systems by using patterns such as graceful degradation, change management, health checks, self‑healing, failover caching, retry logic, rate limiting, circuit breakers, and testing failures to minimize downtime and improve reliability.
Learn essential techniques for designing fault‑tolerant microservices, including graceful degradation, change management, health checks, self‑healing, failover caching, retry strategies, rate limiting, circuit breakers, and testing failures, to ensure high availability and reliability in distributed cloud‑native systems.
This article explains how BIGO’s backbone network achieves high availability through a three‑layer design—control‑plane HA using ETCD‑based Raft leader election, data‑plane HA with MPLS SR‑Policy and intermediate Route‑Reflection layers, and business‑level HA that combines traffic, optimization, and fault scheduling to ensure seamless service continuity.
This article provides a comprehensive guide to Hystrix, covering its purpose in microservice fault tolerance, the problems it addresses, core concepts like command pattern and isolation, detailed workflow steps, configuration options, and practical Java code examples for circuit breaking, fallback, and thread‑pool or semaphore isolation.
This article explains how to build highly available microservice systems by addressing the risks of distributed architectures, employing graceful degradation, change management, health checks, self‑healing, failover caching, retry and rate‑limiting strategies, bulkhead and circuit‑breaker patterns, and continuous failure testing.
This guide explains what RAID is, its role in server storage, compares common RAID levels (0, 1, 5, 6, 10) in terms of fault tolerance, performance, and capacity, and offers recommendations for selecting the most suitable RAID configuration based on data safety, speed, and cost considerations.
This article explains how to build highly available microservice systems by applying fault‑tolerance patterns such as graceful degradation, health checks, self‑healing, failover caches, retries, rate limiting, bulkhead isolation, circuit breakers, and systematic failure testing, while also covering change‑management and deployment strategies.
The article examines B‑Station’s July 2021 outage, explains the concept and quantitative metrics of high availability, and outlines practical techniques such as rate limiting, isolation, failover, timeout control, circuit breaking, degradation, and multi‑region deployment to achieve resilient systems.
This article explains how to build highly available microservices by addressing the inherent risks of distributed systems and presenting fault‑tolerance patterns such as graceful degradation, change management, health checks, self‑healing, failover caching, retries, rate limiting, bulkheads, circuit breakers, and systematic failure testing.
The article explains how to build highly available microservice systems by isolating failures, applying graceful degradation, change‑management, health checks, self‑healing, fallback caches, circuit breakers, retry policies, rate limiting and testing strategies, while acknowledging the cost and operational complexity involved.
This comprehensive guide explains why distributed locks are needed, outlines their three essential properties, compares common implementations such as Redis, MySQL, ZooKeeper, and Redlock, discusses pitfalls like non‑atomic operations and lock expiration, and presents correct patterns using atomic commands, Lua scripts, watchdogs, and fencing tokens.
Reflecting on reading, the author reviews CAP theory’s consistency‑availability‑partition trade‑offs, extends ACID and BASE concepts, proposes modernizing CAP objects to consistency, fault and disaster tolerance, and examines how cloud‑native architectures, micro‑services, and SLA‑driven designs reshape fault tolerance and future self‑healing systems.
The article explains how Spark Streaming's checkpoint mechanism works, detailing the four-step process—from setting the checkpoint directory to writing RDD data and finalizing the checkpoint—highlighting its role in ensuring fault‑tolerant, fast recovery for real‑time recommendation feature pipelines.
This article explains how to build a Docker‑containerized PaaS platform that uses Spring Cloud Netflix components such as Zuul, Eureka, and Hystrix to provide service gateway, registration, discovery, fault tolerance, and dynamic configuration for scalable microservice applications.
This article explains how microservice architectures can achieve high availability by using fault‑tolerant designs such as graceful degradation, health checks, failover caching, circuit breakers, bulkheads, rate limiting, and systematic change‑management practices to mitigate network, hardware, and application errors.
This article explains the Paxos consensus algorithm—its origins, core concepts, roles of proposers, acceptors and learners, safety and liveness constraints, the two-phase protocol, proposal generation, and practical variations—showing why Paxos remains a foundational solution for fault‑tolerant distributed systems.
This article details Ctrip's comprehensive redesign of its airline ticket order database, covering the background performance bottlenecks, vertical and hot‑cold data splitting, sharding key selection, multi‑level caching, cross‑shard query optimization, dual‑write mechanisms, fault‑tolerance strategies, project planning, and the resulting improvements in scalability and cost.
This article explains how to build highly available microservice systems by defining clear service boundaries, employing graceful degradation, change‑management strategies, health checks, self‑healing, cache failover, retry logic, rate limiting, bulkheads, circuit breakers, and testing techniques to mitigate failures in distributed environments.
The article explains how to design resilient distributed systems by minimizing inter‑component dependencies, duplicating or denormalizing data, isolating failures with SLAs, protecting callers and callees, and adding buffers such as asynchronous messaging and elastic scaling to handle random faults as systems grow.
This article explains the concepts of disaster tolerance, fault tolerance, and disaster recovery, compares them with backup purposes, discusses RTO/RPO metrics, investment considerations, and outlines common disaster‑recovery architectures for enterprise IT operations.
The article explains why distributed systems need fault‑tolerance mechanisms such as rate limiting, circuit breaking, and service degradation, describes common metrics (TPS, HPS, QPS), outlines several limiting algorithms (counter, sliding window, leaky bucket, token bucket, distributed and Hystrix‑based), and discusses circuit‑breaker states, considerations, and practical Hystrix usage.
This article explains the risks of microservice architectures and presents a collection of reliability patterns—including graceful degradation, change management, health checks, self‑healing, failover caching, retries, rate limiting, bulkheads, and circuit breakers—to help engineers design and operate highly available backend services.
The article analyzes the 2012 Azure outage caused by a leap‑year date bug, explains Azure's cluster and Fabric Controller architecture, discusses common leap‑year and leap‑second pitfalls, and shows how time anomalies can cascade through DNS and other cloud services, illustrated with real code examples.
This article explains how to design and implement a microservice‑based PaaS platform using Docker containers, Spring Cloud Netflix components such as Zuul, Eureka, and Hystrix, covering service gateway routing, registration and discovery, deployment, fault tolerance, and dynamic configuration.
This article provides a comprehensive guide to microservice monitoring—including log, tracing, and metrics approaches—fault‑tolerance isolation techniques, access‑security mechanisms such as API‑gateway and OAuth2.0, and the role of container technologies like Docker in cloud‑native deployments.
XGBoost‑Ray provides a fault‑tolerant, multi‑node, multi‑GPU backend for XGBoost that integrates seamlessly with Ray Tune, supports distributed data loading, and can be enabled with only three code changes, enabling scalable training and inference on large clusters.
Designing a distributed system involves overcoming major challenges such as heterogeneity, transparency, openness, concurrency, security, scalability, and fault tolerance, each of which must be addressed to build a reliable, extensible, and performant system.
This article explains how to use the Go gobreaker library to implement circuit‑breaker patterns, describing its three states, state transitions, configurable parameters, and providing full source‑code examples to help developers improve fault tolerance in micro‑service architectures.
This article provides a thorough overview of distributed systems, explaining their definition, core concepts such as partition and replication, key challenges, essential characteristics, typical components and protocols, a practical request flow example, and a curated list of real‑world implementations to help readers build a solid learning roadmap.
This article explains the problem of service avalanche in distributed systems, introduces the Hystrix circuit‑breaker concept and its four command modes, evaluates existing PHP implementations, and details the design and implementation of a custom hystrix‑ex Composer package that integrates with Guzzle middleware for high‑concurrency fault tolerance.
This article explores how to design a cloud‑native distributed database architecture by examining TiDB’s current structure, proposing a storage‑compute separation that leverages cloud services like S3 and EBS, and discussing implications for cost, scalability, fault‑tolerance, and multi‑tenant deployment.
This article explains Flink's state concepts, the distinction between keyed and operator state, available state backends, TTL configuration, the mechanics of checkpoints and savepoints, and the two‑phase commit protocol for ensuring exactly‑once processing in streaming applications.
This article thoroughly examines how Redis distributed locks work, the safety challenges they face—including deadlocks, lock expiration, and node failures—explores the Redlock algorithm and its controversies, compares Redis with Zookeeper implementations, and offers practical guidelines and best‑practice solutions for reliable distributed locking.
This article examines the true meaning of consistency in stream computing, clarifies common misconceptions about exactly‑once semantics, formalizes consistency challenges, and reviews how major stream engines such as Google MillWheel, Apache Flink, Kafka Streams, and Spark Streaming implement end‑to‑end consistency.
This article provides a comprehensive overview of distributed system architecture, covering design patterns such as gateways, sidecars and service meshes, performance techniques like caching and sharding, fault‑tolerance mechanisms including rate limiting and circuit breakers, and DevOps practices for deployment and monitoring, all aimed at building resilient cloud‑native applications.
This talk shares practical high‑availability strategies learned from years of building Taobao’s massive e‑commerce platform and migrating to Alibaba Cloud, covering traditional IDC stability mechanisms, cache and disaster‑recovery designs, cloud‑native fault‑tolerance, capacity planning, rate‑limiting, graceful degradation, and multi‑region resilience.
This article explains what distributed locks are, outlines their essential properties, walks through step‑by‑step Redis implementations—from simple SETNX to Lua‑based atomic operations—and discusses reliability strategies such as master‑slave failover and RedLock while highlighting the inherent limits of any distributed lock.
The article details Meituan's five‑year evolution of its instant logistics platform, describing the distributed high‑concurrency architecture, AI‑driven optimization, scalability and fault‑tolerance techniques, and future challenges in micro‑service and unit‑based operations.
This talk shares practical high‑availability design experiences from Alibaba’s e‑commerce platform to its cloud services, covering traditional IDC stability mechanisms, cache and disaster‑recovery strategies, cloud‑native fault handling, capacity planning, traffic shaping, and lessons learned from real incidents.
This talk shares practical high‑availability strategies learned from Alibaba’s Taobao platform and Alibaba Cloud, covering traditional IDC stability, cache and disaster‑recovery designs, cloud‑native fault‑tolerance, performance‑capacity trade‑offs, traffic shaping, multi‑region replication, and lessons from real‑world incidents like GitLab failures.
This article explains the concepts of disaster tolerance, fault tolerance, and disaster recovery, compares them with backup strategies, outlines key metrics such as RTO and RPO, and presents common architectures and planning considerations for building resilient enterprise systems.
This article details Meituan's five‑year journey building a high‑availability, low‑latency instant logistics platform, describing the distributed architecture evolution, AI‑driven optimizations, fault‑tolerance techniques, and future challenges in scaling micro‑services for massive order and rider volumes.
This article explains the circuit breaker pattern, its three-state lifecycle, and provides a detailed Go implementation with code examples, state transition logic, metric tracking, and known limitations for improving service stability.