Facebook migrated a massive, multi‑stage Hive‑based entity ranking pipeline to a single Spark job, detailing the challenges of scaling to 20 TB inputs, the reliability fixes, performance optimizations, and the resulting 4‑6× CPU speedup and reduced latency.
Facebook replaced a multi‑stage Hive pipeline for real‑time entity ranking with a single Spark job, applying extensive reliability fixes and performance tweaks that reduced CPU usage by up to six times, cut latency fivefold, and demonstrated the feasibility of shuffling over 90 TB of data in production.
Alibaba’s three‑year etcd journey reveals how hardware upgrades, software patches, a custom storage freelist algorithm, client best practices, and an enhanced operator platform collectively boosted etcd’s performance 24‑fold, expanded storage 50‑times, and hardened stability for massive Double‑11 workloads.
This article explains how to detect and prevent message loss, guarantee reliable delivery across production, storage, and consumption stages, handle duplicate messages with idempotent designs, resolve message backlogs, and implement distributed transactions using transactional messages in systems like Kafka, RocketMQ, and RabbitMQ.
The article explains Chaos Engineering as a discipline for deliberately injecting failures into distributed systems to uncover hidden weaknesses, outlines its five core principles, describes practical implementation steps and real‑world examples from Ctrip, and discusses future directions for reliability engineering.
This article walks through constructing a lightweight instant‑messaging backend, covering version 1.0.0 features, reliability guarantees, application‑level ACK handling, security encryption, database schema for users, relations and offline messages, and storage strategies to prevent duplicate delivery.
This article presents a comprehensive overview of Netflix's chaos engineering journey, detailing its origins, the development of the Simian Army tools, core principles, practical steps, and applications in Kubernetes environments, offering valuable insights for reliable DevOps practices.
This article explains the challenges of modern large‑scale operations, introduces the end‑state architectural principle, details the system components and safety model, discusses real‑world deployment issues, and looks ahead to future AIOps possibilities, offering practical guidance for building resilient operation platforms.
This article explains the fundamental flaws of typical alert systems, distinguishes between business rule and reliability monitoring, outlines essential metrics and strategies for effective alerts, and presents simple yet powerful anomaly‑detection algorithms to ensure alerts are actionable and reduce noise.
In this detailed talk, Didi senior engineer Du Huan explains the challenges of building large microservice frameworks, outlines design principles such as the Rule of Least Power, describes the evolution of service frameworks, and shares concrete implementation techniques and business benefits of Didi's Go‑based platform.
In his Go China talk, senior Didi engineer Du Huan outlined the design and implementation of a large‑scale microservice framework that abstracts I/O, injects tracing via protocol hijacking, optimizes timers, and enforces fail‑fast circuit breaking, delivering faster development, higher stability, seamless upgrades, and a unified operating‑system‑like layer for thousands of services.
The article explains how contract testing and chaos engineering can be combined to improve the quality and resilience of microservice systems, describing their principles, practical tools such as Chaos Monkey and ChaosBlade, and detailed experiment steps for validating service reliability in cloud‑native environments.
This article presents the goals, methods, and evolution of Qunar's operations team in reducing application failures, improving reliability through fault definition, rapid repair, automation, and AIOps-driven fault prediction, while sharing lessons from industrial PHM and outlining future challenges.
The article explores the concept of trustworthy software, outlines its five key dimensions—safety, reliability, availability, security, and resilience—and describes how Huawei Cloud DevCloud is applying these principles through its open‑source mirror services and secure development practices.
This article explains that high‑availability systems involve more than just redundant hardware or software; they require careful design, data consistency strategies, realistic SLA calculations, and disciplined engineering practices to achieve the coveted “nines” of uptime.
This article explains the essential conditions for TCP communication, details the three‑step handshake and four‑step termination processes with packet‑capture illustrations, and discusses why these sequences ensure reliable connections between a client and a server.
Alibaba’s MonkeyKing, inspired by Netflix’s Chaos Monkey, employs intentional fault injection—from random node kills to simulated network outages—to test and improve system robustness across IaaS, PaaS, and SaaS layers, offering a comprehensive model for reliability engineering in complex distributed environments.
This article explains why chaos engineering is essential for distributed systems, distinguishes it from traditional fault testing, outlines practical inputs and principles, and details Alibaba's multi‑year experience with fault‑drill platforms, automation, and future plans to improve system reliability.
This article explains chaos engineering, its origins at Netflix, core principles, practical steps for running experiments, and how organizations can use controlled failure injection to improve system resilience and operational confidence in complex distributed environments.
This article examines the role of message‑queue middleware in high‑concurrency IM systems, compares popular open‑source options such as ActiveMQ, RabbitMQ, Kafka, RocketMQ and ZeroMQ, and provides a detailed multi‑dimensional framework—including functionality, performance, reliability, operational management, and ecosystem factors—to help engineers select the most suitable queue for their specific business needs.
The article explains how to achieve precise, real‑time detection of physical server outages, reduce false alarms through heartbeat monitoring, network and special‑case interference filtering, and detailed analysis, ultimately improving detection accuracy and coverage for reliable operations.
This article explains how to design comprehensive exception test cases by analyzing four dimensions—business requirements, operational requirements, standard requirements, and experience requirements—to ensure software robustness, reliability, and proper error handling before release.
This article discusses the concept of test right‑shift—moving testing to post‑release production—by detailing a four‑layer online monitoring system for a DSP advertising platform, including interface‑level, UI‑level, revenue, and daily key‑metric monitoring, and shares real‑world incident examples.
Meituan Delivery’s architecture has progressed from a rapid MVP with coarse services to a scalable, fine‑grained platform comprising fulfillment, operation, and master‑data subsystems, employing reliability engineering, capacity planning, AI‑driven simulation, and location services to ensure high availability, efficiency, and future‑ready scalability.
Flash memory’s high performance comes with concerns about reliability, write endurance, and failure rates, so this article explains why flash cells have limited erase cycles, how ECC and LDPC error‑correction extend SSD lifespan, compares SSD vs HDD performance, and outlines factors influencing SSD durability and data recovery.
The article presents practical strategies for avoiding service failures by treating third‑party dependencies as unreliable, designing robust APIs for consumers, and applying solid engineering principles such as degradation plans, timeout settings, traffic control, and resource‑limiting techniques.
The article explores reliability engineering with redundant systems, explains Gödel’s incompleteness theorem and the halting problem, and introduces program static analysis techniques, illustrating how theoretical foundations guide practical approaches to detecting software defects through approximations and abstract interpretation.
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
The article explains how a DSP advertising team moved testing right by building a four‑layer online monitoring system—including interface, UI, revenue, and daily key‑metric monitoring—to quickly detect and resolve production incidents, reduce false alarms, and improve overall reliability.
This article reviews the essential concepts of message‑queue middleware, covering why they are needed, their drawbacks, how to choose among popular solutions, and practical techniques for ensuring high availability, avoiding duplicate consumption, guaranteeing reliable delivery, and preserving message order.
Chaos engineering is a disciplined practice that injects controlled faults into distributed systems—often in production—to validate steady-state hypotheses, uncover hidden reliability weaknesses, and continuously improve resilience, as illustrated by the staged implementations and fault-injection techniques used by companies such as JD.com, Youzan, and Netflix.
To support Meituan’s rapid growth, this article examines the design and evolution of several industry-leading message-queue solutions—including Alibaba’s Notify and RocketMQ, Tencent’s Tube and Hippo, and Ctrip’s Herms—highlighting their reliability, scalability, and decoupling features, and extracting key insights for building robust MQ systems.
This article compares the architectures of Kafka and RocketMQ, examines their availability and reliability mechanisms, evaluates their strengths and weaknesses, and proposes hybrid designs and simplified MQ architectures for building highly available and reliable messaging systems.
The Mi Master billing platform, a SaaS service from Tencent that processes over 735 billion RMB quarterly, provides a unified, modular architecture with distributed multi‑master databases, high‑availability cross‑region deployment, multi‑stage fraud detection, and global support for 80+ payment channels across 180+ countries, delivering seamless APIs, automated reconciliation, and extensive revenue‑sharing tools for products such as Honor of Kings, PUBG, WeChat Pay, and QQ Wallet.
This article reviews essential concepts of message queue middleware, covering why to use it, its drawbacks, selection criteria, high‑availability designs, preventing duplicate consumption, ensuring reliable transmission, and maintaining message order, providing a concise study guide for developers and architects.
This article explains why TCP requires a three‑way handshake to establish connections and a four‑step termination process, using a relatable video‑call scenario and detailed protocol diagrams to illustrate the underlying mechanisms and practical rules for confirming audio transmission.
The article examines the reliability challenges of microservice architectures that rely heavily on synchronous RPC calls, and proposes a comprehensive solution that converts failing RPCs to asynchronous message‑queue workflows, introduces a write‑ahead‑queue for transactional consistency between databases and queues, and outlines offset management to ensure end‑to‑end fault tolerance.
This article demystifies reliability by clarifying the difference between durability and availability, exposing common misconceptions about MTBF, analyzing real‑world disk failure data, and presenting a practical formula for calculating the health probability of distributed storage systems.
This comprehensive guide compares Kafka and RabbitMQ across functional features, performance, reliability, operational management, and ecosystem support, offering practical criteria and pitfalls to help engineers select the most suitable message middleware for their distributed systems.
This article examines Baidu’s Cluster Control System (CCS), detailing its two‑level data model, four‑tier scheduling architecture, and three‑layer execution agents, and explains how control and execution information, redundancy, and fault‑tolerant designs enable reliable large‑scale command execution across thousands of servers.
This article explains the fundamental differences between synchronous and asynchronous communication, explores how message queues decouple producers and consumers, and discusses key considerations such as persistence, performance, reliability, and language support for building robust backend systems.
The article explains Site Reliability Engineering (SRE) as a discipline that blends software engineering with operations, detailing its origins, key responsibilities, required skill sets, tools, impact on reliability and downtime costs, and how the role has evolved with modern cloud and DevOps practices.
In programming and operations, every element—from services and dependencies to requests, machines, data centers, power, networks, and humans—can fail unexpectedly, so you must assume distrust and implement defensive measures such as monitoring, redundancy, rate limiting, fallback strategies, backups, and automated deployment.
This article offers a systematic analysis of product manager core competencies—including product design, requirement documentation, accessibility, reliability, globalization, positioning, and user feedback—while sharing practical insights drawn from the author's ten‑year industry experience and a recent forced break for reflection.
JD’s self‑constructed data center in Suqian, Jiangsu, combines innovative free‑cooling, high‑efficiency chillers, redundant power architecture, and advanced monitoring to deliver over 180 days of natural cooling, reduce PUE to ≤1.3, and ensure continuous operation with robust backup systems.
The article explains how a transaction system evolved from a monolithic application to a service‑oriented design, tackling issues such as RPC‑induced coupling, state explosion, and distributed consistency by introducing reliable event‑driven mechanisms, a core StoreEngine, and an ActorEngine framework.
This article examines the architectures of Kafka and RocketMQ, analyzes their availability and reliability mechanisms, compares their strengths and weaknesses, and proposes hybrid designs and simplified MQ solutions for building robust message‑queue systems.
This article explains Kafka's distributed architecture, its topic‑partition storage model, replication and synchronization mechanisms, reliability guarantees such as ISR and high‑watermark, and presents benchmark results that illustrate how replication factor, acks settings, and partition count affect throughput and latency.
This article explains the role of TCP within the Internet protocol stack, details packet size limits, sequence numbering, assembly, slow‑start and ACK mechanisms, and how TCP ensures reliable, ordered data transmission despite packet loss.
Expanding IoT deployments pressure telecom operators with reliability, security, privacy, and system‑design challenges, requiring large, geographically dispersed networks, robust engineering, and coordinated legal and managerial frameworks to support diverse, large‑scale applications across industries.
This article explains the concept of Site Reliability Engineering (SRE), its origins at Google, core responsibilities such as IT operations and availability improvement, required skill sets, how it differs from DevOps, and guidance on adopting SRE practices within organizations.
An insightful reflection on Google’s SRE philosophy shows how dialectical thinking—questioning absolute stability, embracing limited toil, prioritizing simple monitoring, recognizing automation’s hidden risks, and practicing real‑world failure drills—can reshape operations, encouraging smarter, more resilient system design.
The article shares personal insights from reading Google’s SRE book, explaining core SRE concepts, Google’s robust infrastructure, the role of SLOs, and how they help balance cost, reliability, and innovation in modern operations.
Tencent Cloud CMQ is a Raft‑based distributed message queue that delivers high reliability, strong consistency, and horizontal scalability for finance‑grade workloads, using multi‑node broker sets with majority‑acknowledged writes, automatic leader election, unlimited buffering, full‑path tracing, while requiring application‑level idempotency and offering limited strict ordering.
The article shares a DBA’s three‑year journey at Ganji, detailing core responsibilities, painful incidents like accidental table deletions and massive Redis growth, and practical lessons on stability, backup, hardware prioritization, business alignment, and improving communication between operations and development teams.
The article reflects on the GitLab database deletion incident, analyzing how human error, decision fatigue, inadequate backup strategies, and insufficient safeguards exposed reliability gaps, and proposes practical DevOps practices—such as pair operations, diversified redundancy, strict command restrictions, and continuous feedback—to strengthen complex software systems.
This article explains what high availability means for distributed systems, outlines common availability tiers, and describes how redundancy, load balancing, and automatic failover across a typical Internet architecture can achieve reliable, scalable services.
The article shares practical strategies for preventing service failures by doubting third‑party services, protecting against misuse by consumers, and improving one’s own code and architecture, covering fallback plans, timeout settings, retry policies, API design, traffic control, and resource limits.
This article provides a comprehensive introduction to TCP, covering its header structure, optional fields, the role of MSS and TSO, mechanisms for reliability and retransmission, as well as detailed explanations of flow control, sliding windows, and congestion control algorithms.
The article explains how reliability of storage, servers, and distributed systems is assessed using standards, models like MTBF/MTTR, RAS features, CAP/BASE theories, and end‑to‑end solutions, emphasizing the gap between theoretical metrics and real‑world operational data.
This article examines how backup software ensures enterprise data reliability through media redundancy, server failover, load balancing, and both cold and high‑availability solutions for the management server, highlighting technologies such as GridStor, dual‑array clustering, and deduplication.
The article explains why and when to use message queues, then walks through designing one from scratch, covering decoupling, eventual consistency, broadcast, flow control, RPC protocols, high availability, storage choices, consumer relationships, reliable delivery, transactions, performance optimizations, and push versus pull models.
The article outlines how to design a custom message queue by first identifying appropriate use‑cases such as decoupling, eventual consistency, broadcasting and peak‑shaving, then examining push versus pull models, high‑availability, ordering, duplicate handling, storage choices, batch processing, flow‑control and performance optimizations, with advanced topics reserved for a follow‑up.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
This article explains RAID technology, detailing each RAID level's design, performance trade‑offs, and reliability characteristics so readers can choose the most suitable configuration for their storage needs.
The article shares a former Google SRE’s insights on building high‑availability systems, explaining key factors such as MTBF and MTTR, redundancy strategies like N+2, change‑management practices, and practical tips for reliability engineering and operations.
The article details how LinkedIn has scaled Kafka from handling billions to trillions of messages daily, describing quota enforcement, a ZooKeeper‑free consumer, reliability enhancements, security plans, monitoring frameworks, fault‑injection testing, cluster balancing, and integration with other internal data systems.
Meizu moved from a single‑datacenter to a multi‑datacenter architecture to improve reliability, reduce latency, and meet user proximity demands, detailing technical challenges, traffic scheduling, read‑heavy and read‑write balanced services, and GSLB‑based routing solutions.
The article details how LinkedIn has massively scaled Kafka usage over several years, addressing quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, fault testing, and ecosystem integrations to support its massive data‑driven operations.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
The article reveals how operating‑tool systems, often underestimated, demand high technical rigor to ensure automation success rates and absolute reliability for emergency actions, requiring sophisticated failure handling, capacity awareness, and scalable design—challenges comparable to core online services.
A recent study of five major open‑source distributed systems reveals that most failures can be triggered and reproduced with simple, multi‑event tests, highlighting the importance of systematic testing, deterministic error handling, and concise logging for reliable system operation.
From the evolution of single‑machine file systems to today’s distributed, erasure‑coded cloud storage, this article examines why storage has become increasingly complex, the limitations of traditional replication, and how Qiniu’s next‑gen architecture leverages EC, faster repairs, and cost reductions to meet scalability, reliability, and availability demands.
