This article explains how Hystrix protects distributed Java applications from cascading failures by using circuit breakers, thread‑pool or semaphore isolation, timeout detection, fallback strategies, and health statistics, and includes key source code excerpts illustrating each mechanism.
The article analyzes Bilibili’s recent service outage, explains why high availability matters, introduces key metrics like MTBF and MTTR, and outlines practical strategies such as redundancy, rate limiting, isolation, failover, timeout control, circuit breaking, degradation, and multi‑region deployment to build resilient systems.
Analyzing Hystrix’s source code reveals how its circuit‑breaker, bulkhead isolation (semaphore or thread‑pool), timeout detection, fallback logic, and sliding‑window health metrics work together to prevent cascading failures in distributed systems, as illustrated by an e‑commerce order service calling multiple downstream services.
This article thoroughly examines the safety of Redis‑based distributed locks, explains basic SETNX locking, explores deadlock and lock‑release problems, presents robust solutions such as atomic SET with expiration, Lua scripts, and unique tokens, and critically compares Redlock with Zookeeper while summarizing expert debates and best‑practice recommendations.
The article explains three frequent pitfalls when adopting microservices—complex remote communication, asynchronous processing challenges, and distributed transaction difficulties—and shows how fast‑fail, retries, timeouts, compensation, lightweight workflow engines, and idempotency can reduce complexity and improve resilience.
This article details Baidu’s internal service‑mesh deployment, explaining why traditional RPC‑based governance fell short, how a sidecar‑based mesh decouples governance from frameworks, and the technical challenges and solutions for low‑intrusion, high‑performance, fault‑tolerant traffic management across tens of thousands of microservices.
Baidu created an internally‑built, Istio‑based service mesh that decouples governance from language‑specific RPCs, offering low‑intrusion integration, ultra‑low latency via a brpc coroutine data plane, advanced fault‑tolerance and fine‑grained traffic scheduling, and now powers over 80 % of its core microservices handling more than a trillion daily requests.
This article explores essential patterns and techniques—such as graceful degradation, change management, health checks, failover caching, retry logic, rate limiting, circuit breakers, and chaos testing—to build highly available microservice architectures that can withstand network, hardware, and application failures.
Gaode Ride‑Hailing created a comprehensive 360° observability platform—standardized logging, distributed tracing, multi‑domain metrics, visual dashboards, and an incident workflow—that transforms raw data into actionable insights, accelerates root‑cause analysis, and enables automated fault defense for its large‑scale cloud‑native microservice system.
Microservice architectures must handle inevitable network, hardware, and application errors by employing fault‑tolerant patterns such as graceful degradation, change management, health checks, fail‑over caches, retry logic, rate limiting, circuit breakers, and testing strategies to maintain service reliability and user experience.
This article describes the challenges of native Redis in multi‑data‑center deployments and presents the design, implementation, and performance evaluation of a custom Redis extension that adds bidirectional synchronization, rlog logging, protocol enhancements, and conflict‑resolution mechanisms to achieve reliable cross‑region active‑active operation.
This article explains high availability fundamentals, defines availability, outlines design targets, presents common metrics such as MTBF, MTTR, MTTF, SA, RPO, RTO, discusses CAP theory, essential design elements, and answers practical Q&A on cost, architecture, fault tolerance, testing, and implementation guidance.
Kafka partitions are the fundamental storage units that enable topics to scale horizontally, maintain message order within each partition, provide fault‑tolerance through replication, and support parallel consumption, with various write strategies such as key‑based, round‑robin, and custom rules.
This article chronicles the evolution of an online supermarket from a simple monolithic website to a fully split microservice architecture, discussing the motivations, challenges, design patterns, monitoring, fault tolerance, testing, service discovery, and the eventual adoption of service mesh.
Raft is a fault‑tolerant distributed consensus protocol that simplifies Paxos by electing a single leader each term to coordinate client requests, replicate logs to a majority of servers, ensure safety through up‑to‑date voting, handle failures with randomized timeouts, resolve log conflicts, and compress logs via snapshots.
This article details the design of a red packet calculation and distribution system for a Spring Festival promotion, addressing mixed-type packet computation, seamless transition to awarding, distributed task processing, stability guarantees, and successful implementation results.
Spring Festivalbatch processingdistributed computing
This article describes how to design and build a native distributed cluster called MyCluster without using any existing frameworks, covering master‑slave architecture, leader election, split‑brain handling, centralized configuration management, custom communication protocols, state transitions, and client interfaces.
This article explains how Apache Flink has evolved into a unified stream‑batch engine by introducing a three‑layer architecture, a unified DataStream SDK, a pipeline‑region‑based DAG scheduler, a common shuffle framework, and enhanced fault‑tolerance mechanisms to address efficiency, consistency, and resource‑utilisation challenges in real‑time big‑data processing.
This article details Baidu's Feed recommendation system architecture, explaining how a combination of dynamic retry scheduling, real‑time stop‑loss mechanisms, multi‑recall frameworks, ranking layer fallbacks, and IDC‑level multi‑master designs collectively ensure five‑nine availability across billions of daily requests.
This article explains why distributed theory and algorithms are crucial for architects, outlines the four foundational theories and eight key protocols, discusses their four evaluation dimensions, and provides a step‑by‑step learning roadmap illustrated with stories and practical examples.
The article details Meituan's five‑year journey in instant logistics, describing how distributed, high‑concurrency backend architectures were progressively upgraded to microservices, how AI is integrated for pricing, ETA and dispatch, and the operational techniques used to ensure scalability, fault tolerance, and high availability.
The paper presents a cluster‑aware Guava cache component for Alibaba’s Xianyu platform that mitigates downstream service failures by adding asynchronous reload, cluster‑wide key invalidation, and size reporting, enabling automatic fallback to refreshed local data and improving latency, with future plans for a management console, tiered storage, and disk‑backed caching.
This article explains Dubbo’s internal architecture, covering service registration and discovery with Zookeeper, RPC invocation details including load balancing, routing, and fault‑tolerance strategies, as well as its network protocol and thread‑dispatch mechanisms, providing practical insights for backend developers.
This article explains how to build a microservice‑based PaaS platform using Spring Cloud Netflix components, Docker containers, Eureka for service registration, Zuul as a gateway, Hystrix for fault tolerance, and a dynamic configuration center to achieve agile development and continuous integration.
This article explains why ZooKeeper clusters should use an odd number of nodes, how the majority quorum mechanism avoids split‑brain scenarios, and outlines practical solutions such as quorums, redundant communication, fencing, arbitration, and disk‑lock techniques to ensure reliable distributed coordination.
Designing a distributed system involves overcoming major challenges such as heterogeneity, transparency, openness, concurrency, security, scalability, and fault tolerance, each requiring careful consideration of hardware, software, network, and management aspects to build robust, scalable, and secure architectures.
The article explains that disaster recovery for information systems relies on redundancy across hardware, energy, and data, classifies natural, human, and technical disasters, defines critical metrics such as RTO and RPO, and outlines the technologies, architectures, and maturity levels needed to ensure business continuity.
This article explains how to implement distributed locks using Redis, covering the essential requirements of mutual exclusion, deadlock avoidance, and fault tolerance, detailing single‑instance and multi‑instance algorithms, code examples with SETNX and Lua scripts, and discussing challenges such as latency, crashes, and persistence.
This article explains how Manbang built a rigorous stability‑assurance framework—including strict fault grading, a "watch‑and‑protect" system, blue‑green deployments, online pressure testing, fault‑drill platforms, and runtime metadata—to ensure rapid iteration while maintaining high availability for millions of logistics users.
The article thoroughly explains Kafka’s message‑queue reliability design and implementation, covering use‑case scenarios, core concepts, storage format, producer acknowledgment settings, broker replication mechanisms (ISR, HW, LEO), consumer delivery semantics, the epoch solution for synchronization, and practical configuration guidelines for various consistency and availability requirements.
This article explains why Redis clusters are needed, how CLUSTER MEET builds the network, slot assignment, scaling procedures, client redirection mechanisms, batch operations, fault detection, recovery processes, and common operational pitfalls, providing practical guidance for building and maintaining a robust Redis Cluster deployment.
The article explains how Netflix uses Chaos Monkey and a suite of related tools, collectively called the Simian Army, to deliberately inject failures into their cloud infrastructure, continuously test fault‑tolerance, and ensure high availability and reliability for their streaming service.
Microservice architectures increase system complexity and failure rates, so this article explains key reliability patterns—such as graceful degradation, change management, health checks, self‑healing, fallback caches, retry logic, rate limiting, circuit breakers, and testing—to help engineers design resilient, high‑availability services.
This article examines the inherent risks of microservice architectures and presents practical patterns—such as graceful degradation, change management, health checks, self‑healing, fallback caching, retries, rate limiting, bulkheads, and circuit breakers—to build highly available, fault‑tolerant services.
This article explains how to integrate Netflix Hystrix into Spring Cloud applications to provide request timeout, circuit‑breaker, fallback, monitoring and resource isolation for microservice calls, including Maven setup, annotation usage, Feign client fallback configuration and disabling options.
This article explains why microservice calls need fault‑tolerance mechanisms, introduces Hystrix’s core features such as timeouts, circuit‑breaker, fallback, monitoring and resource isolation, and provides step‑by‑step code examples for integrating Hystrix and Feign in a Spring Cloud project.
This article describes ByteDance's single‑task recovery solution for Flink's real‑time computation, detailing the problem of global job restarts, the proposed network‑layer enhancements, upstream and downstream optimizations, JobManager restart strategy, implementation challenges, and the measurable latency and availability benefits achieved in production.
The article presents a design for an asynchronous component that is monitorable, fault‑tolerant, and integrates with zero overhead, compares Akka, RxJava, and a custom JUC‑based implementation, and selects the latter—using extended Callables and a CountDownLatch—to track business units, handle timeouts, and provide fallback behavior.
The article introduces Tars, an open‑source high‑performance RPC framework and integrated service governance platform derived from Tencent's internal microservice architecture, detailing its design philosophy, layered architecture, core features such as the Tars protocol, load balancing, fault and overload protection, and centralized configuration management.
This article explains how concurrency issues that appear in single‑threaded programs become amplified in distributed systems, covering consistency models, network reliability, clock synchronization, fault detection, backpressure, and cascading failures, and offers practical design and testing strategies to build resilient architectures.
This article explains how to use Redis and Redisson for distributed locking, introduces an abstract DistributedLock interface for flexible implementations, demonstrates automatic lock release with functional callbacks, and discusses fallback strategies and monitoring to ensure reliability in backend systems.
This article explains the concept of distributed architecture, its evolution from monolithic systems, core design principles, common challenges such as network latency and data consistency, and how major tech companies adopt it to achieve high availability, scalability, and fault tolerance.
Distributed architecture, built on distributed computing technologies like J2EE, transforms monolithic systems into multi‑layer, fault‑tolerant platforms by decoupling services, ensuring high availability, scalability, and resilience, while addressing challenges such as network latency, data consistency, and system complexity, as illustrated by real‑world case studies.
This article describes how Qunar scaled its IDC infrastructure, introduced multi‑IDC deployment, automated DNS‑based load balancing, open‑source DNSDB, and an IDC proxy built on Squid to achieve rapid fault self‑healing and transparent traffic switching for both user and third‑party access.
This article provides a thorough overview of distributed systems, explaining their definition, core challenges, key characteristics, essential components, common protocols, and practical implementations to help readers build a solid, structured learning path for mastering distributed architectures.
The article explains that Dubbo services remain callable after the registry fails because consumers cache provider metadata locally, the registry cluster can failover automatically, and the framework’s robustness ensures continued operation despite various component outages.
This article explains the Byzantine Generals Problem, maps its concepts to distributed consensus, distinguishes consensus from consistency, outlines oral‑message and signed‑message solutions, and analyzes their applicability and limitations in fault‑tolerant distributed systems.
This article presents a comprehensive distributed retry system that leverages a distributed scheduling mechanism to ensure eventual consistency, reduce manual recovery costs, and provide flexible retry strategies, automatic recovery detection, visual management, rate limiting, and intelligent retry for backend services.
This article explains the core concepts of replication, consistency, and fault tolerance in distributed systems, discusses strong and asynchronous replication methods, and details the CAP theorem with its consistency, availability, and partition tolerance trade‑offs, illustrating AP and CP scenarios such as Eureka and Zookeeper clusters.
This article outlines how to create solid software architectures by presenting a process for forming design principles, detailing fifteen universal architecture guidelines, and explaining service‑splitting and key design rules that together help build scalable, maintainable, and resilient systems.
This article explains how Netflix's Hystrix framework provides thread‑pool and semaphore isolation, circuit breaking, and fallback mechanisms to prevent cascading failures in distributed systems, offering practical code examples and deployment strategies for building resilient backend services.
This article explains disaster tolerance, fault tolerance, and disaster recovery, compares them with backup, discusses RTO/RPO metrics, outlines disaster types, and presents common disaster‑recovery architectures and planning considerations for enterprise IT operations.
This article provides a comprehensive overview of Apache Flink, explaining how it unifies stream and batch processing on a single runtime, detailing its key features, APIs, libraries, architectural components, fault‑tolerance mechanisms, scheduling, iterative processing, and back‑pressure monitoring.
This article provides a thorough introduction to the Raft consensus algorithm, explaining its purpose, core components such as state machine replication, log and consensus module, leader‑follower model, client interaction, fault‑tolerance considerations, the CAP trade‑off, and why Go is a suitable implementation language.
The article thoroughly explains Apache Flink’s checkpoint mechanism—including state types, coordinator workflow, exactly‑once versus at‑least‑once semantics, common failure sources such as code exceptions, storage or network issues, and practical configuration tips like interval settings, local recovery and externalized checkpoints.
This article provides a comprehensive overview of distributed system fundamentals, covering node modeling, replica concepts, consistency levels, data distribution strategies, centralized and decentralized replica protocols, lease mechanisms, quorum, two‑phase commit, MVCC, Paxos, and the CAP theorem, while analyzing their trade‑offs in availability, consistency, and partition tolerance.
This article walks through the transformation of an online supermarket from a simple monolithic website to a fully fledged microservice architecture, highlighting the motivations, design decisions, common pitfalls, and essential components such as monitoring, tracing, logging, gateways, service discovery, circuit breaking, testing strategies, and service mesh adoption.
This article compares Spring Cloud and Dubbo across protocol handling, performance tuning, load‑balancing strategies, fault‑tolerance mechanisms, and routing/traffic‑shaping features, highlighting their flexibility, configuration complexity, and suitability for different microservice scenarios.
This article details Alibaba's massive Flink deployment, covering its historical background, the operational challenges of managing tens of thousands of nodes, the design of a comprehensive Flink management platform, and the automated solutions for fault handling, resource allocation, and performance testing in a large‑scale big‑data environment.
This article explains three core design principles—consistency, fault tolerance, and readability—showing how they reduce learning costs, prevent user errors, and improve text clarity, ultimately making designs more persuasive and user‑friendly.
This article presents Google SRE expert Ramón Medrano Llamas’s comprehensive guide on designing, operating, and maintaining large‑scale, highly available software systems, covering SRE fundamentals, daily workflows, scalability strategies, fault‑tolerant architecture, monitoring, and operational best practices.
Kafka consumer groups enable multiple consumers to share partition workloads, ensuring exclusive consumption within a group, flexible consumption patterns like broadcast and unicast, and automatic fault‑tolerance through rebalancing, ultimately improving throughput, scalability, and resilience of streaming applications.
The article details eBay's 2018‑2020 design, performance testing, and fault‑tolerance architecture of a next‑generation core accounting system capable of handling millions of transactions per second, covering system goals, multi‑region deployment, event‑sourcing, Raft consensus, scalability optimizations, and the planned open‑source release.
This article explains why Spark checkpoints are needed for large or complex RDD pipelines, how they work by persisting data to reliable storage such as HDFS, and outlines practical steps and best‑practice recommendations for using checkpoints effectively in production environments.
This article explains the definitions of disaster tolerance, fault tolerance, and disaster recovery, compares their purposes, discusses backup versus disaster‑tolerance solutions, outlines key metrics such as RTO and RPO, and presents common architectural and investment considerations for building resilient enterprise systems.
This article provides a comprehensive overview of distributed system architecture, covering essential design patterns such as gateways, sidecars, and service meshes, performance techniques like caching and async communication, fault‑tolerance mechanisms including rate limiting and circuit breakers, and practical DevOps practices for deployment and monitoring.
The article describes how a high‑traffic payment platform achieves 99.999% availability by avoiding single points of failure, applying fail‑fast principles, implementing resource limits, building real‑time monitoring and alerting, and automating fault detection, routing, and recovery to ensure continuous 7×24 operation.
High Availabilitybackend operationsfault tolerance
This article presents a detailed, step‑by‑step illustration of microservice architecture, covering its motivations, component breakdown, migration from monoliths, common pitfalls, monitoring, tracing, logging, gateway, service discovery, resilience patterns, testing strategies, frameworks, and the emerging service‑mesh approach.
This article explains Flink's checkpoint mechanism, its execution flow, common configuration options, and the benefits and considerations of incremental checkpoints using the RocksDB state backend, providing practical code examples and YAML settings for reliable stream processing.
ElasticDL, the first industry‑level open‑source system for elastic deep learning on TensorFlow, leverages Kubernetes‑native scheduling, fault‑tolerance, and TensorFlow 2.0 Eager Execution to dramatically improve cluster utilization, simplify distributed training, and integrate seamlessly with tools like Kubeflow and SQLFlow.
This article explores the two toughest problems in distributed stream processing—exactly‑once message handling and ordering—by dissecting the underlying impossibility of perfect failure detectors, the liveness‑vs‑safety trade‑off, zombie processes, and the practical solutions employed by systems such as Flink, Kafka Streams, MillWheel, and Spark.
This article explains the design and implementation of Spark Structured Streaming's StateStore module, covering its distributed architecture, state sharding, versioning, batch read/write, migration, update/query APIs, maintenance compaction, and fault‑tolerance mechanisms that enable incremental continuous queries with exactly‑once guarantees.
This article explains the Paxos consensus algorithm in plain terms, using a relatable travel‑planning analogy to illustrate how proposers, acceptors, and majority voting achieve fault‑tolerant agreement in distributed systems, and connects the concept to real‑world implementations like Google’s Chubby and ZooKeeper.
QTSDB is a distributed time‑series database built on InfluxDB that adds cluster capabilities such as horizontal scaling, replica fault‑tolerance, sharding, meta‑service coordination, and comprehensive read/write processing, while also detailing failure recovery and node expansion strategies.
This article explains the concept of fault tolerance in software architecture, illustrates why thread‑pool isolation is essential for preventing cascading failures, and provides concrete Java implementations—including code examples, pros and cons, and practical guidance for applying the technique in real‑world backend systems.
EasyScheduler is an open‑source big‑data workflow scheduler that uses a decentralized architecture with Master and Worker nodes coordinated via ZooKeeper, supporting DAG‑based task definitions, various task types, fault tolerance, priority handling, distributed locks, and remote logging, all illustrated with detailed component diagrams.
This article introduces EasyScheduler, an open‑source big data workflow scheduling system, explaining its core terminology, decentralized architecture, distributed lock implementation, thread‑shortage handling, fault‑tolerance mechanisms, task‑retry and priority designs, as well as its logging solution using Logback and gRPC.
This article explains the essential requirements for achieving 99.99% service availability—consistency, eliminating single points, placement groups, traffic isolation, same‑city active‑active, N+1 redundancy, and multi‑region active‑active—illustrated with a step‑by‑step Yum repository service case study and evolving architecture diagrams.
The QLive service powers iQIYI’s massive live‑streaming events—such as the Spring Festival Gala—by combining vertical and horizontal scaling, a three‑layer architecture with dual data‑center isolation, multi‑level caching, circuit‑breaker/degradation controls, and a Flume‑Kafka‑Hive monitoring pipeline to sustain over 400 k QPS and 99.9999 % availability.
This article explains Apache Flink's fault‑tolerance mechanisms, including checkpointing, barrier alignment, the differences between At‑Least‑Once and Exactly‑Once semantics, configuration options, incremental checkpointing, and the requirements for external sources and sinks to achieve end‑to‑end exactly‑once processing.
This article explains why Hystrix is essential for large distributed systems, describes its dependency isolation mechanisms—including command, group, thread‑pool, and semaphore isolation—covers circuit‑breaker behavior, fallback strategies, and provides detailed Java code examples for configuration and usage.
The article explains service isolation in system architecture, its origins, why it matters, two main isolation approaches (by service and by user), their advantages and drawbacks, and key considerations to ensure fault containment and improve overall system availability.
This article examines multiple fault‑tolerant transaction designs—RSM‑based KV, RSM‑based transactions, shared‑storage approaches, high‑availability KV layers, and single‑node engine extensions—comparing their replication strategies, lock handling, and performance trade‑offs while raising open questions about ordering and consistency.
This article examines three frequent pitfalls encountered when integrating microservices—complex communication, asynchronous challenges, and distributed transaction difficulties—and proposes mitigation techniques such as rapid failure handling, workflow engines, timeout management, and compensation patterns to improve resilience and reduce system complexity.
This article explains the concept of circuit breaking in distributed systems, outlines a four‑step implementation process with strategies for detecting unhealthy services, cutting off calls, probing recovery, and restoring normal operation, and shares best‑practice tips to minimize downtime and improve resilience.
The article shares a comprehensive, experience‑driven guide on building fault‑tolerant systems—covering retry mechanisms, dynamic node removal, timeout settings, service degradation, decoupling, and business‑level safeguards—to enable a platform that scales from millions to billions of daily requests without relying on manual fire‑fighting.
This article analyzes the challenges of implementing a generic TCC distributed transaction framework using Spring, emphasizing the need for RM‑local transaction integration, transaction manager takeover, fault recovery, idempotency, and proper handling of Try/Confirm/Cancel operations.
This article shares practical strategies for avoiding system failures by doubting third‑party services, protecting against misuse by consumers, and strengthening internal design through solid API practices, resource limits, and disciplined coding principles.
This article uses a vivid allegorical story to introduce the Paxos algorithm, then explains its roles, two-phase protocol, fault assumptions, and why majority and multiple acceptors are essential for achieving reliable consensus in distributed systems.
The article explains what constitutes a fault, introduces key metrics such as RPO and RTO, and describes various fault isolation principles, patterns, and practical examples—including dependency degradation, failover, dynamic adjustment, fast‑fail, caching, rate limiting, and resource isolation—to improve system reliability.
The article explains how the rapid growth of big‑data technologies reshapes human cognition by shifting from sample‑based, precise analysis to whole‑population, fault‑tolerant, correlational, and ultimately intelligent thinking that mirrors the human brain.
The article explains the concept of service isolation, its origins in shipbuilding, why it’s crucial for reducing fault impact in software systems, practical approaches such as functional and user‑based isolation, their trade‑offs, and key design principles to ensure reliable, maintainable back‑end architectures.
This article compares Spark Streaming and Flink across runtime models, component roles, programming APIs, task scheduling, time semantics, dynamic Kafka partition detection, fault‑tolerance mechanisms, exactly‑once guarantees, and back‑pressure handling, providing code examples and practical insights for real‑time data processing.
This article explains how microservices use service registries for discovery, registration, health checks, and deregistration, compares third‑party and self‑registration, explores server‑side and client‑side call mechanisms, discusses API gateways, synchronous vs asynchronous messaging, and outlines fault‑tolerance patterns such as timeouts, circuit breakers, and bulkheads.
The article presents practical strategies for avoiding service failures by treating third‑party dependencies as unreliable, designing robust APIs for consumers, and applying solid engineering principles such as degradation plans, timeout settings, traffic control, and resource‑limiting techniques.
This article explains how to construct a PaaS cloud platform using a microservice architecture, Spring Cloud Netflix components, Docker containers, and supporting services such as a gateway, Eureka registration, Hystrix fault tolerance, and a dynamic configuration center.
This article explains how to construct a PaaS cloud platform using microservice architecture and Docker containers, detailing the roles of Spring Cloud Netflix components such as Zuul, Eureka, Hystrix, and Config Server, and covering gateway routing, service discovery, deployment, fault tolerance, and dynamic configuration.
Meituan/Dianping ensures its critical account service stays online by combining real‑time business monitoring, circuit‑breaker‑driven graceful degradation, and active‑active cross‑region deployment with isolated dependencies, versioned data sync, and automated cache updates, dramatically extending MTBF while cutting MTTR and latency.
High Availabilitydata synchronizationfault tolerance
This article outlines guiding principles, high‑availability strategies, and high‑concurrency techniques—covering stateless design, resource isolation, quota management, monitoring, degradation, rollback, and scaling—to help engineers build resilient, scalable systems while balancing cost and performance.
This article examines common hardware and network failures in data centers, analyzes real‑world outage cases, classifies fault domains, and presents comprehensive strategies for database fault handling—including logging, checkpointing, backup, replication, and high‑availability architectures—to improve reliability and reduce downtime.