This article discusses the concept of test right‑shift—moving testing to post‑release production—by detailing a four‑layer online monitoring system for a DSP advertising platform, including interface‑level, UI‑level, revenue, and daily key‑metric monitoring, and shares real‑world incident examples.
Meituan Delivery’s architecture has progressed from a rapid MVP with coarse services to a scalable, fine‑grained platform comprising fulfillment, operation, and master‑data subsystems, employing reliability engineering, capacity planning, AI‑driven simulation, and location services to ensure high availability, efficiency, and future‑ready scalability.
Flash memory’s high performance comes with concerns about reliability, write endurance, and failure rates, so this article explains why flash cells have limited erase cycles, how ECC and LDPC error‑correction extend SSD lifespan, compares SSD vs HDD performance, and outlines factors influencing SSD durability and data recovery.
The article presents practical strategies for avoiding service failures by treating third‑party dependencies as unreliable, designing robust APIs for consumers, and applying solid engineering principles such as degradation plans, timeout settings, traffic control, and resource‑limiting techniques.
The article explores reliability engineering with redundant systems, explains Gödel’s incompleteness theorem and the halting problem, and introduces program static analysis techniques, illustrating how theoretical foundations guide practical approaches to detecting software defects through approximations and abstract interpretation.
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
The article explains how a DSP advertising team moved testing right by building a four‑layer online monitoring system—including interface, UI, revenue, and daily key‑metric monitoring—to quickly detect and resolve production incidents, reduce false alarms, and improve overall reliability.
This article reviews the essential concepts of message‑queue middleware, covering why they are needed, their drawbacks, how to choose among popular solutions, and practical techniques for ensuring high availability, avoiding duplicate consumption, guaranteeing reliable delivery, and preserving message order.
Chaos engineering is a disciplined practice that injects controlled faults into distributed systems—often in production—to validate steady-state hypotheses, uncover hidden reliability weaknesses, and continuously improve resilience, as illustrated by the staged implementations and fault-injection techniques used by companies such as JD.com, Youzan, and Netflix.
To support Meituan’s rapid growth, this article examines the design and evolution of several industry-leading message-queue solutions—including Alibaba’s Notify and RocketMQ, Tencent’s Tube and Hippo, and Ctrip’s Herms—highlighting their reliability, scalability, and decoupling features, and extracting key insights for building robust MQ systems.
This article compares the architectures of Kafka and RocketMQ, examines their availability and reliability mechanisms, evaluates their strengths and weaknesses, and proposes hybrid designs and simplified MQ architectures for building highly available and reliable messaging systems.
The Mi Master billing platform, a SaaS service from Tencent that processes over 735 billion RMB quarterly, provides a unified, modular architecture with distributed multi‑master databases, high‑availability cross‑region deployment, multi‑stage fraud detection, and global support for 80+ payment channels across 180+ countries, delivering seamless APIs, automated reconciliation, and extensive revenue‑sharing tools for products such as Honor of Kings, PUBG, WeChat Pay, and QQ Wallet.
This article reviews essential concepts of message queue middleware, covering why to use it, its drawbacks, selection criteria, high‑availability designs, preventing duplicate consumption, ensuring reliable transmission, and maintaining message order, providing a concise study guide for developers and architects.
This article explains why TCP requires a three‑way handshake to establish connections and a four‑step termination process, using a relatable video‑call scenario and detailed protocol diagrams to illustrate the underlying mechanisms and practical rules for confirming audio transmission.
The article examines the reliability challenges of microservice architectures that rely heavily on synchronous RPC calls, and proposes a comprehensive solution that converts failing RPCs to asynchronous message‑queue workflows, introduces a write‑ahead‑queue for transactional consistency between databases and queues, and outlines offset management to ensure end‑to‑end fault tolerance.
This article demystifies reliability by clarifying the difference between durability and availability, exposing common misconceptions about MTBF, analyzing real‑world disk failure data, and presenting a practical formula for calculating the health probability of distributed storage systems.
This comprehensive guide compares Kafka and RabbitMQ across functional features, performance, reliability, operational management, and ecosystem support, offering practical criteria and pitfalls to help engineers select the most suitable message middleware for their distributed systems.
This article examines Baidu’s Cluster Control System (CCS), detailing its two‑level data model, four‑tier scheduling architecture, and three‑layer execution agents, and explains how control and execution information, redundancy, and fault‑tolerant designs enable reliable large‑scale command execution across thousands of servers.
This article explains the fundamental differences between synchronous and asynchronous communication, explores how message queues decouple producers and consumers, and discusses key considerations such as persistence, performance, reliability, and language support for building robust backend systems.
The article explains Site Reliability Engineering (SRE) as a discipline that blends software engineering with operations, detailing its origins, key responsibilities, required skill sets, tools, impact on reliability and downtime costs, and how the role has evolved with modern cloud and DevOps practices.
In programming and operations, every element—from services and dependencies to requests, machines, data centers, power, networks, and humans—can fail unexpectedly, so you must assume distrust and implement defensive measures such as monitoring, redundancy, rate limiting, fallback strategies, backups, and automated deployment.
This article offers a systematic analysis of product manager core competencies—including product design, requirement documentation, accessibility, reliability, globalization, positioning, and user feedback—while sharing practical insights drawn from the author's ten‑year industry experience and a recent forced break for reflection.
JD’s self‑constructed data center in Suqian, Jiangsu, combines innovative free‑cooling, high‑efficiency chillers, redundant power architecture, and advanced monitoring to deliver over 180 days of natural cooling, reduce PUE to ≤1.3, and ensure continuous operation with robust backup systems.
The article explains how a transaction system evolved from a monolithic application to a service‑oriented design, tackling issues such as RPC‑induced coupling, state explosion, and distributed consistency by introducing reliable event‑driven mechanisms, a core StoreEngine, and an ActorEngine framework.
This article examines the architectures of Kafka and RocketMQ, analyzes their availability and reliability mechanisms, compares their strengths and weaknesses, and proposes hybrid designs and simplified MQ solutions for building robust message‑queue systems.
This article explains Kafka's distributed architecture, its topic‑partition storage model, replication and synchronization mechanisms, reliability guarantees such as ISR and high‑watermark, and presents benchmark results that illustrate how replication factor, acks settings, and partition count affect throughput and latency.
This article explains the role of TCP within the Internet protocol stack, details packet size limits, sequence numbering, assembly, slow‑start and ACK mechanisms, and how TCP ensures reliable, ordered data transmission despite packet loss.
Expanding IoT deployments pressure telecom operators with reliability, security, privacy, and system‑design challenges, requiring large, geographically dispersed networks, robust engineering, and coordinated legal and managerial frameworks to support diverse, large‑scale applications across industries.
This article explains the concept of Site Reliability Engineering (SRE), its origins at Google, core responsibilities such as IT operations and availability improvement, required skill sets, how it differs from DevOps, and guidance on adopting SRE practices within organizations.
An insightful reflection on Google’s SRE philosophy shows how dialectical thinking—questioning absolute stability, embracing limited toil, prioritizing simple monitoring, recognizing automation’s hidden risks, and practicing real‑world failure drills—can reshape operations, encouraging smarter, more resilient system design.
The article shares personal insights from reading Google’s SRE book, explaining core SRE concepts, Google’s robust infrastructure, the role of SLOs, and how they help balance cost, reliability, and innovation in modern operations.
Tencent Cloud CMQ is a Raft‑based distributed message queue that delivers high reliability, strong consistency, and horizontal scalability for finance‑grade workloads, using multi‑node broker sets with majority‑acknowledged writes, automatic leader election, unlimited buffering, full‑path tracing, while requiring application‑level idempotency and offering limited strict ordering.
The article shares a DBA’s three‑year journey at Ganji, detailing core responsibilities, painful incidents like accidental table deletions and massive Redis growth, and practical lessons on stability, backup, hardware prioritization, business alignment, and improving communication between operations and development teams.
The article reflects on the GitLab database deletion incident, analyzing how human error, decision fatigue, inadequate backup strategies, and insufficient safeguards exposed reliability gaps, and proposes practical DevOps practices—such as pair operations, diversified redundancy, strict command restrictions, and continuous feedback—to strengthen complex software systems.
This article explains what high availability means for distributed systems, outlines common availability tiers, and describes how redundancy, load balancing, and automatic failover across a typical Internet architecture can achieve reliable, scalable services.
The article shares practical strategies for preventing service failures by doubting third‑party services, protecting against misuse by consumers, and improving one’s own code and architecture, covering fallback plans, timeout settings, retry policies, API design, traffic control, and resource limits.
This article provides a comprehensive introduction to TCP, covering its header structure, optional fields, the role of MSS and TSO, mechanisms for reliability and retransmission, as well as detailed explanations of flow control, sliding windows, and congestion control algorithms.
The article explains how reliability of storage, servers, and distributed systems is assessed using standards, models like MTBF/MTTR, RAS features, CAP/BASE theories, and end‑to‑end solutions, emphasizing the gap between theoretical metrics and real‑world operational data.
This article examines how backup software ensures enterprise data reliability through media redundancy, server failover, load balancing, and both cold and high‑availability solutions for the management server, highlighting technologies such as GridStor, dual‑array clustering, and deduplication.
The article explains why and when to use message queues, then walks through designing one from scratch, covering decoupling, eventual consistency, broadcast, flow control, RPC protocols, high availability, storage choices, consumer relationships, reliable delivery, transactions, performance optimizations, and push versus pull models.
The article outlines how to design a custom message queue by first identifying appropriate use‑cases such as decoupling, eventual consistency, broadcasting and peak‑shaving, then examining push versus pull models, high‑availability, ordering, duplicate handling, storage choices, batch processing, flow‑control and performance optimizations, with advanced topics reserved for a follow‑up.
LinkedIn’s engineering team details how they have scaled Apache Kafka from billions to over a trillion daily messages, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, and ecosystem integrations to improve cost, availability, and performance.
LinkedIn’s engineering team details how they scaled Kafka from a few billion to over a trillion daily messages, covering quotas, a new ZooKeeper‑free consumer, reliability upgrades, security roadmaps, monitoring frameworks, failure testing, cluster balancing, and ecosystem integrations.
This article explains RAID technology, detailing each RAID level's design, performance trade‑offs, and reliability characteristics so readers can choose the most suitable configuration for their storage needs.
The article shares a former Google SRE’s insights on building high‑availability systems, explaining key factors such as MTBF and MTTR, redundancy strategies like N+2, change‑management practices, and practical tips for reliability engineering and operations.
The article details how LinkedIn has scaled Kafka from handling billions to trillions of messages daily, describing quota enforcement, a ZooKeeper‑free consumer, reliability enhancements, security plans, monitoring frameworks, fault‑injection testing, cluster balancing, and integration with other internal data systems.
Meizu moved from a single‑datacenter to a multi‑datacenter architecture to improve reliability, reduce latency, and meet user proximity demands, detailing technical challenges, traffic scheduling, read‑heavy and read‑write balanced services, and GSLB‑based routing solutions.
The article details how LinkedIn has massively scaled Kafka usage over several years, addressing quotas, a new ZooKeeper‑free consumer, reliability enhancements, security features, monitoring frameworks, fault testing, and ecosystem integrations to support its massive data‑driven operations.
Since 2011, LinkedIn has expanded its Kafka deployment from handling billions to over a trillion messages per day, focusing on quotas, a new ZooKeeper‑free consumer, reliability enhancements, security, monitoring frameworks, fault‑injection testing, cluster balancing, and ecosystem integrations, offering valuable lessons for large‑scale streaming systems.
The article reveals how operating‑tool systems, often underestimated, demand high technical rigor to ensure automation success rates and absolute reliability for emergency actions, requiring sophisticated failure handling, capacity awareness, and scalable design—challenges comparable to core online services.
A recent study of five major open‑source distributed systems reveals that most failures can be triggered and reproduced with simple, multi‑event tests, highlighting the importance of systematic testing, deterministic error handling, and concise logging for reliable system operation.
From the evolution of single‑machine file systems to today’s distributed, erasure‑coded cloud storage, this article examines why storage has become increasingly complex, the limitations of traditional replication, and how Qiniu’s next‑gen architecture leverages EC, faster repairs, and cost reductions to meet scalability, reliability, and availability demands.
