This article explains how Conway's law guides functional team division in microservice architectures, describes decentralized governance, outlines various interaction and composition patterns, discusses fault‑tolerance mechanisms such as isolation, circuit breaking, rate limiting, and provides guidance on choosing appropriate service granularity.
The article compares Raft, a crash‑only consensus used in private blockchains with leader election and O(N) communication, to PBFT, a Byzantine‑tolerant protocol for consortium chains that requires 3f+1 nodes, uses three‑phase messaging, O(N²) complexity, and different fault‑tolerance limits.
This article explains how setting low timeouts for non‑core services, decoupling and physically isolating micro‑services, separating light and heavy workloads, and implementing automated configuration checks together enhance system reliability and reduce both technical and human errors in high‑traffic environments.
This article examines the pitfalls of naive retry mechanisms, explores active‑standby service switching, dynamic removal of unhealthy nodes, proper timeout configuration, and anti‑reentrancy strategies to improve system reliability and prevent cascading failures in large‑scale backend operations.
This article explains service branch mode, its impact on availability, asynchronous messaging, shared‑data anti‑patterns, and essential microservice fault‑tolerance techniques such as compartment isolation, circuit breaking, rate limiting, and failover strategies.
Effective fault tolerance is crucial for microservice stability, and this article explores core design principles and classic patterns—such as timeout retries, rate limiting, bulkhead isolation, circuit breakers, and fallback strategies—guiding developers to choose and combine the right approaches for high‑availability systems.
This article reflects on ten years of Tencent's operations experience, sharing the author's career journey, the evolution of large‑scale service management, the design of the L5 fault‑tolerant system, unified frameworks, resource packaging, CMDB virtual mirrors, and automated deployment practices that together enable reliable, efficient, and scalable infrastructure.
This article details the redesign of a TV app from a horizontal layout to a waterfall flow, explaining the project timeline, advantages, new seven‑layer architecture, CMS‑driven configuration, compatibility handling, pagination strategies, caching, and fault‑tolerance measures that together improved user conversion and system robustness.
Veteran Tencent operations leader Zhao Jianchun recounts a decade of managing 100,000 servers, detailing the L5 fault‑tolerant system, unified framework, resource packaging, CMDB virtual imaging, and an automated deployment platform that together cut daily incidents by up to 90% and boosted efficiency tenfold.
This article explains how Tencent's ZhiYun full‑link log monitoring platform handles massive daily logs, overcomes challenges of diverse log formats, high throughput, fault‑tolerant design, and provides scalable storage, query, and alerting capabilities for distributed micro‑service environments.
This article explains why high availability is critical for internet services, outlines key techniques such as stateless design, service discovery, heartbeat checks, idempotent operations, load balancing, throttling, caching, and micro‑service architecture, and discusses the operational challenges and monitoring tools needed to maintain resilient, scalable systems.
This article outlines essential high‑concurrency and high‑availability principles—including stateless design, service decomposition, caching strategies, message queues, data heterogeneity, degradation, rate limiting, traffic switching, and rollback mechanisms—to help architects build scalable, reliable, and resilient systems.
In programming and operations, every element—from services and dependencies to requests, machines, data centers, power, networks, and humans—can fail unexpectedly, so you must assume distrust and implement defensive measures such as monitoring, redundancy, rate limiting, fallback strategies, backups, and automated deployment.
Alibaba’s eight‑year evolution of Double 11 showcases how limited cost can deliver maximal user experience and massive throughput by transitioning from a centralized 3.0 distributed architecture to multi‑active zones, employing capacity planning, full‑link stress testing, fine‑grained dependency governance, and dynamic traffic scheduling to ensure high availability.
This article presents a comprehensive overview of fault governance in large‑scale distributed systems, covering classic dependency failures, the concept of strong and weak dependencies, experimental observations, the evolution of fault‑injection techniques, and best practices for building reliable fault‑drill platforms.
This article explains Redis Cluster’s distributed architecture, the 16,384 hash slot mechanism, node addition and removal, decentralised request routing, fault‑tolerance through master‑slave replication, slot migration without service interruption, and the internal data structures that enable these features.
PacificA is Microsoft’s generic replication framework for large‑scale distributed storage systems that provides strong consistency, separates configuration management from data replication, and uses a primary‑secondary model with lease‑based fault detection to ensure availability, correctness, and efficient operation across heterogeneous nodes.
This article examines Apache Flink's stream‑processing architecture, compares its native streaming model, fault‑tolerance, performance and SQL capabilities with Spark and Storm, and concludes that Flink offers a more powerful and efficient solution despite some maturity gaps.
This article recounts Alibaba's middleware team's QCon Beijing 2017 presentation on fault governance and fault‑drill practices, covering distributed‑system dependency failures, strong/weak dependency concepts, multi‑stage technical evolution, and the design of their chaos‑engineering platform for large‑scale e‑commerce.
Microservices architecture introduces technical concerns such as service registration, discovery, load balancing, health checks, front-end routing, fault tolerance, dynamic configuration, and framework selection, with common solutions ranging from centralized and in-process load balancers to Netflix and Spring Cloud components.
In his ASPLOS keynote, Alibaba’s Vice President Zhou Jingren detailed the company’s large‑scale stream and graph computing platforms, highlighting fault‑tolerance innovations, real‑time data challenges, and upcoming advances in graph analytics and massive machine‑learning workloads.
The article reviews hardware and software load‑balancing, explains classic strategies such as round‑robin, random, minimum‑response‑time, least‑connections and hash, and quantitatively evaluates their fault‑tolerance using probability formulas and example scenarios in distributed systems.
Least Connectionsdistributed systemsfault tolerance
This article explains microservice fault‑tolerance techniques—including timeout settings, retry strategies, circuit‑breaker logic, current limiting, resource isolation, and service degradation—from both micro and macro perspectives, illustrating how to design resilient service chains and avoid cascading failures.
TDSQL’s cloud‑based audit solution combines a three‑proxy high‑availability layer, Kafka’s O(1) persistent messaging, and a distributed audit‑server that uses consistent hashing and multi‑coroutine processing to consume data within seconds, while fault‑tolerant offsets, majority acknowledgments, and Tencent Cloud MongoDB storage ensure secure, ordered, scalable, and highly reliable audit logging.
The article details the QQ Game Spring Festival Red Packet backend, describing its multi‑phase architecture that handles 80 k RPS, uses CDN‑served static gift data, two‑level sorting, CMEM caching, RocketMQ buffering for throttled delivery, idempotent order tracking for fault tolerance, and unified real‑time monitoring.
The article shares Tencent’s practical fault‑tolerance journey for a billion‑scale activity platform, covering retry strategies, automatic removal of faulty nodes, timeout tuning, business‑level safeguards, service degradation, and decoupling techniques that together reduce manual firefighting and improve system resilience.
JIMDB is JD’s in‑house distributed cache platform that combines automatic fault detection, seamless online scaling, multi‑language support, and containerized deployment to replace traditional Memcached/Redis solutions, offering features such as one‑click cluster creation, elastic expansion, lossless scaling, and comprehensive monitoring for high‑traffic e‑commerce services.
This article explains why microservices are chosen over monolithic architectures, outlines their key characteristics, reviews popular frameworks such as Netflix's stack and Spring Cloud, and discusses essential concerns like service discovery, load balancing, security, configuration, monitoring, and fault tolerance.
Disque is an experimental, distributed, fault‑tolerant in‑memory message queue built in C that extends Redis concepts with synchronous replication, configurable delivery semantics, explicit acknowledgments, fast‑ack support, dead‑letter handling, and optional disk persistence for robust backend messaging workloads.
The article explains how Meituan‑Dianping applies classic fault‑tolerance patterns—timeout and retry, rate limiting/load shedding, circuit breaker, bulkhead isolation, and fallback—to design for failure, prevent cascading service outages, and enhance system stability and high‑availability in a service‑oriented architecture.
This article explains how HDFS, inspired by Google’s GFS, provides a low‑cost, highly reliable, fault‑tolerant, and high‑performance distributed file system for big‑data workloads by using replication, standby NameNodes, block storage, rack awareness, and compute‑close‑to‑data strategies.
This article provides a vivid explanation of the Paxos algorithm, illustrating how it achieves reliable consensus among unreliable processors through a two‑phase prepare/promise and propose/accept process, using distributed auction analogies, message sequencing, and read/write operations to ensure consistency in distributed systems.
The article presents a business‑centric high‑availability solution for a large‑scale game access platform, detailing measurable goals, a three‑dimensional architecture that includes client‑side retry, HTTP‑DNS, functional separation, multi‑region active‑active deployment, and automated, visual monitoring to achieve rapid problem detection, recovery, and minimal outage frequency.
business continuitydistributed systemsfault tolerance
This article shares Qunar's architecture and practical experience in designing a self‑developed high‑availability message middleware, covering its role in transaction processing, consistency guarantees, fault‑tolerance mechanisms, isolation, monitoring, and consumer design, and discusses trade‑offs and operational considerations.
This article provides a beginner-friendly, engineering-focused overview of the production‑grade Paxos library PhxPaxos, explaining the consensus protocol, its roles, instance management, state‑machine integration, performance optimizations, multi‑group deployment, and practical considerations such as disk durability, leader election, and log checkpointing.
This article shares the author’s experience building fault‑tolerance for Tencent’s activity operations platform, covering retry strategies, automatic removal of unhealthy machines, timeout tuning, asynchronous processing, anti‑replay mechanisms, service degradation, service decoupling, and business‑level safeguards to reduce manual alarm handling and improve system robustness.
This article shares Tencent’s experience building fault‑tolerant mechanisms for the AMS activity platform, covering retry strategies, automatic machine exclusion, timeout tuning, service isolation, asynchronous processing, anti‑replay safeguards, and operational best practices that transformed a million‑request service into an 800‑million‑request system.
This article examines common load‑balancing algorithms—round‑robin, random, minimum response time, minimum concurrency, and hash—analyzing their fault‑tolerance in distributed clusters, deriving success‑rate formulas, and showing why strategies like minimum concurrency outperform simple methods under node failures.
This article explains what microservices are, compares them with monolithic architecture, outlines their key characteristics, and provides practical guidance on client access, inter‑service communication, service discovery, fault tolerance, and deployment considerations.
This article explains the principles of real-time computing, compares it with offline batch processing, and demonstrates a practical solution using Kafka for ingestion, Apache Storm for continuous computation, and various storage options, while also covering streaming concepts and Storm's high‑availability mechanisms.
The article presents a comprehensive, business‑centric high‑availability architecture for a game access platform, detailing measurable goals, a three‑layered design, client‑side retry with HTTP‑DNS, functional separation and degradation, multi‑region active‑active deployment, and automated, visual monitoring to achieve rapid issue detection, recovery, and minimal downtime.
business reliabilitydistributed systemsfault tolerance
This article systematically explores the distributed characteristics of NoSQL databases, covering data consistency, placement, peer systems, anti‑entropy protocols, eventual consistency data types, sharding, fault detection, and coordinator election, illustrating how these strategies balance scalability, availability, latency, and fault tolerance.
The article explains the need for low‑latency, high‑performance, distributed real‑time processing, outlines the challenges such systems must address, and introduces Storm as a Hadoop‑like framework for stream processing, detailing its architecture, fault‑tolerance mechanisms, transactional topology, and large‑scale deployment at Taobao.
The article explains how distributed machine learning evolved from parallel computing to handle massive, long‑tail data sets, discusses the importance of scalability, fault recovery, and data‑parallel algorithms, and reviews frameworks such as MPI, MapReduce, and Pregel for building large‑scale AI systems.
An experienced backend engineer shares practical strategies to prevent service failures, covering third‑party distrust, user‑side safeguards, robust API design, traffic limiting, resource management, and architectural best practices such as single‑responsibility and avoiding single points of failure.
The article explains the historical background, core characteristics such as scalability, cost‑efficiency, fault tolerance, and elasticity, and outlines essential design principles for distributed systems, emphasizing low hardware requirements, horizontal scaling, avoidance of single points of failure, reduced communication overhead, and stateless services.
Voidbox integrates Docker containers with YARN to simplify distributed application development, improve deployment, boost cluster efficiency, and provide fault‑tolerant, DAG‑based execution modes, enabling seamless resource management for Hadoop‑based big data jobs.
The article explains how an API Gateway can consolidate multiple fine‑grained microservice endpoints into a single coarse‑grained API for mobile clients, discussing its design considerations, benefits, drawbacks, and implementation techniques such as reactive programming, service discovery, and fault tolerance.
The article details Uber senior engineer Zhao Lei’s presentation on building a highly available messaging platform, covering single‑point failure mitigation, sharding approaches, large‑scale outage handling, cross‑region failover, and the practical engineering practices and protocols used to keep billions of users online.
Distributed systems, long used since the 1970s and popularized by internet giants like Google, offer scalable, cost‑effective, and fault‑tolerant architectures by leveraging many low‑cost servers, emphasizing horizontal scaling, avoidance of single points of failure, minimal inter‑node communication, and stateless services for elastic application deployment.
Horizontal ScalingStateless Servicesdistributed systems
The article explains the origins, core features such as scalability, cost‑effectiveness, fault tolerance, and elastic expansion of distributed systems, and outlines design philosophies like low hardware requirements, horizontal scaling, avoidance of single points of failure, minimized inter‑node communication, and stateless service architecture.
Horizontal ScalingStateless Servicesdistributed systems
Disque, an open‑source distributed in‑memory message broker created by Redis's creator, offers synchronous replication, flexible delivery semantics, multi‑master clustering, and fine‑grained job control, but remains in Alpha with several limitations that make it unsuitable for production use today.
This article explains the concept, architecture, and trade‑offs of various RAID levels—including RAID 0, 1, 2, 3, 4, 5, 6, and hybrid configurations—detailing how they combine multiple inexpensive disks to achieve different balances of speed, capacity, and fault tolerance for server environments.