This article explains why introducing a message queue like Kafka helps decouple systems and control traffic, then dives into the three key questions of detecting, locating, and preventing message loss, offering concrete monitoring methods, configuration settings, and troubleshooting steps for producers, brokers, and consumers.
This article explains why payment orders can disappear after a successful wallet transaction, outlines the complete payment flow, distinguishes internal and external drop‑order scenarios, and provides practical server‑side and client‑side strategies—including retry mechanisms, reliable async messaging, scheduled queries, and delayed‑message approaches—to reliably prevent such issues.
This article explains the key challenges of cloud development—data management, design and implementation, and messaging—then presents a detailed catalog of cloud design patterns, each with problem statements, Azure examples, and categories such as reliability, security, and performance efficiency.
This article outlines the core reliability principles of the Google Cloud Architecture Framework, explaining key terms such as SLI, SLO, error budget, and SLA, and describing design and operational guidelines for defining reliability goals, building observability, ensuring high availability, creating robust processes, effective alerting, and collaborative incident management.
This article describes how Qunar Travel has evolved its testing environment governance from a fixed 10‑machine setup to a template‑driven, soft‑routing architecture (Environment 3.0), improving delivery speed, reliability, business connectivity, and reducing operational costs through automated sync, smart recommendations, and continuous business checks.
WLock is an open‑source distributed lock service built on the WPaxos consensus algorithm and RocksDB storage, offering multiple lock types, flexible acquisition modes, high concurrency optimizations, and strong reliability and throughput for coordinating shared resources in distributed systems.
This article introduces the core concepts, components, exchange types, reliability mechanisms, ordering, delay, transaction, high‑availability, and load‑balancing strategies of three popular message‑queue systems—RabbitMQ, RocketMQ, and Kafka—while also discussing common challenges such as message ordering, delayed consumption, reliability, idempotence, and backlog handling.
This article provides an in‑depth technical guide to three major message‑queue systems—RabbitMQ, RocketMQ, and Kafka—covering their core components, exchange and routing mechanisms, durability features, consumer acknowledgment, dead‑letter handling, load balancing, and practical strategies for reliability, ordering, and scaling in distributed backend applications.
This article provides a comprehensive overview of Site Reliability Engineering, explaining its origins, core responsibilities across infrastructure, platform, and business layers, daily tasks such as deployment, on‑call duties, SLI/SLO management, incident post‑mortems, capacity planning, and user support, as well as career advice for aspiring SREs.
Chaos engineering, a discipline for experimenting on distributed systems, helps teams identify hidden weaknesses, improve high‑availability, and build confidence in production by defining stable states, injecting realistic failures, and measuring impact through observability metrics, with practical steps, tool choices, maturity stages, and evaluation methods.
Reliability and resilience are distinct yet complementary goals for cloud services; reliability is the outcome of consistently meeting performance expectations, while resilience describes a system’s ability to continue operating despite failures, and this article introduces the concepts and outlines a four‑part series exploring related threats and enhancement techniques.
This article explains the essential concepts of TCP and UDP, covering the OSI layers, physical and data‑link networking, IP addressing, transport‑layer mechanisms such as sliding windows, congestion control, reliable transmission, connection establishment and termination, and compares the strengths and weaknesses of TCP versus UDP.
This article examines the design principles, challenges, and implementation details of a multi‑active architecture for hybrid cloud environments, covering stability, cost, efficiency, network topology, container orchestration, service discovery, traffic scheduling, and data storage, and outlines practical solutions used by the Zuoyebang platform.
This article explains how TCP ensures reliable connections through the three-way handshake process—including SYN, SYN‑ACK, and ACK exchanges—and describes the four-step termination sequence with FIN and ACK flags, while also clarifying why three handshakes are necessary instead of two.
This article compares Alibaba Cloud's MSE cloud‑native gateway with the traditional Nginx Ingress Controller across performance, cost, reliability and security, presenting benchmark data, CVE analysis, architectural advantages, and a step‑by‑step migration plan for Kubernetes clusters.
This article explains how RocketMQ and RabbitMQ guarantee message reliability by covering production, storage, and consumption stages, introducing confirm mechanisms, persistence settings, manual acknowledgments, and compensation strategies such as message database storage to achieve near‑zero data loss in distributed systems.
This article examines the inherent shortcomings of the TCP protocol, including the difficulty of upgrading the stack, the latency introduced by its three‑way handshake and TLS, head‑of‑line blocking caused by packet loss, and the high cost of connection migration when network conditions change.
Message‑queue middleware such as RabbitMQ, RocketMQ, ActiveMQ, and Kafka introduces challenges like ordering, loss, duplication, back‑pressure and delayed delivery, which can be mitigated by using single‑consumer queues or partitioning, enabling acknowledgments and replication, applying idempotent identifiers, scaling consumers, and employing dead‑letter or scheduling mechanisms.
This article explains the concept of chaos engineering, its six key benefits, the design of a full‑lifecycle chaos platform, fault‑atom categories, experiment orchestration, risk control, automation, red‑blue war games, and practical experiments that helped Tencent Games improve system reliability while cutting operational costs.
The article examines Alibaba Cloud's DNS operation platform, detailing its three‑stage evolution—standardization, automation, and intelligent automation—and how these practices achieve sub‑10 ms latency stability, zero‑downtime fault isolation, and scalable reliability for billions of daily queries.
This article systematically explains the relationships among high concurrency, high availability, performance, and scalability, defines their quantitative metrics, categorizes sources of change that affect system reliability, and outlines strategies for fault prediction, impact reduction, and rapid recovery in large‑scale services.
The article explains how to achieve 100% message delivery and consumption in MQ systems by covering producer acknowledgments, broker persistence mechanisms, and consumer idempotency, with detailed comparisons of RabbitMQ and Kafka implementations and configuration tips.
This article explains the motivations for introducing message queues—such as decoupling, asynchronous processing, and traffic shaping—then details the typical challenges like availability, complexity, duplicate consumption, data consistency, message loss, ordering, and backlog, and provides practical solutions for each issue.
This article provides an in‑depth introduction to Apache Kafka, covering its distributed streaming platform fundamentals, message‑queue models, topic and partition design, broker and cluster roles, producer partitioning logic, reliability guarantees, consumer group assignors, offset management, and performance optimizations such as sequential disk writes and zero‑copy techniques.
This article continues the previous discussion on TCP handshakes and termination, then thoroughly explains nine essential TCP mechanisms—including sliding window, flow control, congestion control, delayed and piggyback ACKs, sticky packet handling, and keep‑alive—illustrated with diagrams and practical examples.
This article explains why message queues are used, outlines common scenarios such as decoupling, asynchronous processing and traffic shaping, compares major MQ products, and provides practical guidance on high availability, preventing loss, duplicate consumption, ordering, backlog handling, and expiration.
This article explains how to guarantee that messages are never lost when using MQ middleware such as Kafka, RabbitMQ, or RocketMQ, outlines the key interview points, and provides practical design patterns, detection mechanisms, idempotency, and scaling strategies for reliable message delivery.
This article explains the typical problems encountered with message queues, such as message loss, duplicate delivery, and distributed transaction handling, and details various solutions including local message tables, MQ‑based transactions, and the specific mechanisms used by RocketMQ, Kafka, and RabbitMQ to ensure reliability and consistency.
This article outlines Ant Group's six‑year journey in chaos engineering, describing its three generational evolutions, business‑oriented fault injection, risk‑mining, full‑lifecycle coverage, massive scale, root‑data protection, core technologies such as Awatch, simulation environments, and plans for intelligent, open‑source future development.
The article shares a comprehensive methodology for MySQL database inspection, classifying checks by method, time and depth, and detailing availability, reliability, and performance best‑practice items along with concrete command examples and configuration recommendations.
This article explains how TCP establishes reliable connections with a three‑way handshake, why three steps are necessary, and how it gracefully closes connections using a four‑way termination, detailing each packet flag and state transition.
This article examines the stability challenges of selecting distributed data products, introduces chaos‑engineering‑based testing methods, outlines practical test scenarios, fault injection techniques, toolchains, and quantitative analysis metrics, and presents a capability assessment standard for ensuring system reliability.
This article provides a comprehensive overview of Kafka, covering its layered architecture with Zookeeper, core concepts such as topics, partitions and consumer groups, communication workflow, partition selection strategies, rebalancing mechanisms, reliability configurations, replica synchronization, and reasons for moving away from Zookeeper, all explained in clear English.
This article explains how TCP establishes reliable connections using a three‑way handshake, why two‑way handshakes are insufficient, and details the four‑step termination process, including the purpose of the TIME‑WAIT state in TCP/IP networks.
This article provides an in‑depth technical guide to Apache Kafka, covering its distributed streaming architecture, core concepts such as topics, partitions, brokers, producers and consumers, reliability guarantees, storage mechanisms, configuration parameters, and consumer assignment strategies, supplemented with Java code examples.
The article explains how to achieve high‑reliability message delivery in RabbitMQ by using producer confirm mode, persisting exchanges, queues and messages, storing outbound messages in a database, and switching consumers to manual acknowledgments to prevent loss in network or broker failures.
Service degradation, a crucial reliability technique, involves selectively disabling non-essential features, applying rate limiting, timeout handling, fallback data, and tiered switches across front‑end, back‑end, and infrastructure layers to maintain core functionality during traffic spikes or component failures, ensuring high availability and meeting SLA targets.
This article shares ten practical operations guidelines—from avoiding duplicated work and embracing mistakes to emphasizing monitoring, backup roles, clear division of labor, and continuous improvement—aimed at boosting reliability, efficiency, and team cohesion for both engineers and managers.
This article provides a comprehensive comparison between RocketMQ and Kafka, covering data reliability, performance, queue capacity, real‑time delivery, retry mechanisms, ordering guarantees, scheduled messages, transactional support, query capabilities, message tracing, consumer parallelism, filtering, and commercial backing, helping engineers choose the right messaging middleware for their workloads.
This article introduces the fundamentals of Site Reliability Engineering (SRE), explains how to measure service stability with metrics like MTTR, MTBF, and availability, outlines the SRE workflow from prevention to post‑mortem, and discusses how to identify and reduce technical debt to improve system health.
The 2021 Accelerate State of DevOps report by Google DORA, based on over 32,000 professionals, reveals that elite teams dramatically outperform low‑performing teams across four classic delivery metrics, introduces a new reliability metric, and highlights the impact of team culture, SRE practices, cloud adoption, secure software supply chains, and high‑quality documentation on software delivery and organizational performance.
This article explains Kafka's storage architecture, including segment files and indexing, and details the reliability mechanisms such as ISR, OSR, LEO, HW, producer acknowledgment levels, and leader election strategies to ensure data consistency and availability.
This article explains HDFS reliability features such as replica policies, rack awareness, heartbeat, safe mode, checksums, trash, metadata protection and snapshots, then details YARN failover handling for ApplicationMaster, NodeManager and ResourceManager, and finally describes the Hadoop MapReduce shuffle workflow and tuning tips.
This article examines the key problems message middleware must address—such as publish/subscribe, ordering, filtering, persistence, reliability, latency, and transaction support—and explains how Apache RocketMQ’s architecture and design choices provide high‑performance, high‑throughput solutions to each of these challenges.
This article explains the fundamentals of the Transmission Control Protocol, its reliable byte‑stream characteristics, the detailed three‑way handshake process, and why three steps are required to prevent duplicate or stale connections in network communication.
This article explains the fundamentals of message queues, compares synchronous and asynchronous communication, discusses the benefits of sender‑receiver decoupling, outlines performance and reliability considerations, and provides practical guidance for designing robust distributed messaging architectures.
A real‑world incident with ActiveMQ’s high‑availability setup shows that focusing on component reliability without business‑level capacity planning, monitoring, and graceful degradation can cripple services, highlighting that true high availability must prioritize overall system and user experience.
Qunar shares how it built a large‑scale chaos engineering platform for thousands of microservices, detailing tool selection, architecture, evolution stages, fault‑injection scenarios, strong/weak dependency automation, open‑source contributions, and future plans for automated random drills.
This article details Baidu Search's high‑availability engineering, describing eight major challenges in fault analysis and the corresponding innovations—index mirroring, streaming analysis, comprehensive label sets, feature engineering, query reconstruction, intelligent ranking, timeline analysis, and chaos engineering—that together enable near‑real‑time, 99% accurate detection and mitigation of search service failures.
This article explains how Kafka can suffer message loss and duplication, and provides practical configurations for the producer, broker, and consumer—including acks, retries, replication factor, min.insync.replicas, and manual offset commits—to achieve reliable, idempotent message processing.
This article provides a comprehensive overview of message middleware concepts, the role and benefits of message queues, and an in‑depth look at RabbitMQ architecture, components, reliability mechanisms, advanced features, and internal storage and flow‑control mechanisms.
This article explains system availability, outlines three stages of incident handling, identifies key factors that degrade availability such as human error, avalanche effects, untested releases and infrastructure failures, and proposes technical and team‑oriented practices to enhance reliability and achieve higher "nines" of uptime.
This article explains the core concepts of message queues—including producers, consumers, messages, brokers, and push/pull delivery—and analyzes three consumption scenarios: at‑most‑once, at‑least‑once, and exactly‑once, detailing the required producer, broker, and consumer behaviors for each.
This article explains the drawbacks of traditional synchronous architectures, outlines why adopting message queues improves latency, coupling, and peak‑handling, and then details common MQ problems such as duplicate messages, data inconsistency, loss, ordering, backlog, and increased complexity along with concrete mitigation strategies.
This article explains how to use chaos engineering with Gremlin to design, execute, and analyze four experiments that test Kafka broker load, message loss, split‑brain scenarios, and ZooKeeper outages, helping improve the reliability and resilience of Kafka deployments.
iQIYI’s Little Deer Chaos Platform injects faults and runs red‑blue attacks across production services, enabling teams to validate alerts, circuit‑breakers, and fail‑over mechanisms—demonstrated by video playback and membership service case studies—thereby fostering zero‑trust design, faster skill growth, and resilient cloud‑native operations.
The article explains how to detect and prevent message loss, guarantee reliable delivery, handle duplicate consumption, manage message backlog, and implement distributed transaction messages in systems like Kafka, RocketMQ, and RabbitMQ, providing practical code examples and best‑practice guidelines.
This article explains practical techniques for detecting lost messages, guaranteeing reliable production, storage, and consumption stages, handling duplicate deliveries with idempotent designs, managing message backlogs, and implementing distributed transactions using transactional messages in modern message queue systems.
This article provides a comprehensive overview of RabbitMQ, covering its AMQP fundamentals, exchange types, reliable delivery mechanisms, consumer flow control, idempotency, dead‑letter queues, TTL, clustering modes, and related high‑availability tools such as HAProxy and Keepalived.
Netflix describes how it extended the academic Molly fault‑injection method into an automated, production‑scale failure‑testing system that explores dependency trees, defines success criteria, runs short low‑impact experiments, and discovers hidden faults before they affect users.
This article explains the fundamentals of version control, why it is essential for DevOps teams, and how it improves productivity, error detection, parallel development, and overall product reliability through better collaboration and traceability.
Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.
The article explains service governance and SRE practices, detailing goals, components, overload handling, capacity planning, and strategies to maintain continuous 24‑hour service reliability while reducing manual toil.
Microservice architectures must handle inevitable network, hardware, and application errors by employing fault‑tolerant patterns such as graceful degradation, change management, health checks, fail‑over caches, retry logic, rate limiting, circuit breakers, and testing strategies to maintain service reliability and user experience.
This article explains high availability in distributed systems, covering its definition, design objectives, key metrics such as MTBF, MTTR, SLA, and practical design elements like redundancy, monitoring, failover, as well as common Q&A on cost, relationship with other architecture attributes, and implementation considerations.
The article examines five different retry mechanisms for ensuring reliable message delivery between payment and billing microservices, evaluates their pros and cons, and ultimately recommends the third solution as a cost‑effective, highly reliable approach achieving 99.99% consistency.
This article explains the architecture, reliability mechanisms, dynamic configuration, monitoring, and alerting strategies of JD Daojia Open Platform's Business Message Queue (BMQ), illustrating how the system handles bidirectional communication, fault isolation, and scalable message processing for merchants.
The article provides a comprehensive overview of Site Reliability Engineering (SRE), explaining its origins, definition by Google, required skill sets, typical responsibilities, tools used, and how the role has evolved within DevOps and modern cloud‑native environments.
This guide presents thirteen practical Redis best‑practice recommendations covering memory optimization, performance tuning, high reliability, routine operations, resource planning, monitoring, and security, offering concrete steps such as key length control, maxmemory configuration, lazy‑free, command selection, connection pooling, replication settings, and safe deployment to help developers and DBA operators run Redis efficiently and safely.
This article explores common pitfalls in RabbitMQ message delivery, explains persistence settings and the confirm mechanism, and why they alone can't ensure 100% reliability, then proposes a robust solution combining pre‑persisting messages to Redis, confirm callbacks, scheduled retries, and idempotent consumer design to achieve near‑zero loss.
This article provides a comprehensive overview of RabbitMQ, covering its AMQP‑based architecture, core components such as exchanges and queues, reliability mechanisms like publisher confirms and dead‑letter queues, consumer flow control, clustering and federation modes, and related high‑availability tools.
This article explains how to integrate RabbitMQ into a Spring Boot application, covering dependency setup, connection configuration, message production and consumption, handling complex JSON messages, and ensuring both sending and receiving reliability through publisher confirms, return callbacks, and consumer acknowledgements.
The article details how Xianyu’s 2020 messaging failures—lost messages, wrong avatars, and order status errors—were traced to duplicate IDs, push‑logic mismatches, and client bugs, and solved by introducing global UUIDs, ACK‑based retries, hierarchical conversation models, hybrid storage caching, and real‑time monitoring, boosting delivery reliability above 99.9%.
This article provides a comprehensive introduction to Apache Kafka, covering its distributed publish‑subscribe architecture, its key components such as brokers, topics, partitions, producers, consumers, and ZooKeeper, as well as its advantages, drawbacks, storage mechanisms, partition assignment strategies, and reliability guarantees for high‑throughput big‑data streaming.
This article explains the twenty‑one key non‑functional characteristics of software architecture—such as performance, reliability, scalability, security and maintainability—detailing their definitions, typical metrics, and practical techniques for improvement, while linking each trait to ISO‑25010 and real‑world engineering practices.
This article examines how to guarantee reliable message delivery in RabbitMQ by using durable queues, the confirm callback mechanism, pre‑persisting messages with Redis or a database, scheduled compensation tasks, and idempotent processing techniques such as optimistic locking and unique‑ID fingerprinting.
This article explains the definitions, calculations, and practical importance of MTTR, MTBF, and MTTF for reliability engineering, showing how accurate data and proper metric use enable SRE teams to improve system availability, plan maintenance, and reduce downtime.
This article explains how to integrate RabbitMQ into a Spring Boot application, covering dependency setup, connection configuration, message sending and receiving, handling complex JSON messages, and ensuring both publishing and consumption reliability through confirmations, callbacks, and QoS settings.
This article explains RabbitMQ's AMQP fundamentals, exchange types, reliable delivery mechanisms, idempotency strategies, consumer flow control, TTL and dead‑letter handling, as well as clustering, federation, HAProxy and Keepalived solutions for building a resilient messaging infrastructure.
This article provides a comprehensive guide to RabbitMQ, covering core AMQP concepts, exchange types, message reliability mechanisms, dead‑letter handling, QoS settings, and various high‑availability cluster modes such as mirrored, federated, and multi‑active deployments.
This article provides a comprehensive overview of RabbitMQ, covering AMQP core concepts, exchange and queue types, message reliability techniques such as confirms and returns, consumer flow‑control, TTL and dead‑letter handling, as well as clustering, federation, and HAProxy/Keepalived high‑availability solutions.
This article explores the challenges of frontend availability, introduces chaos engineering concepts, and details Alibaba's practical approach to frontend fault injection—including static resource hijacking, a safe isolated environment, monitoring integration, and a real‑world drill that demonstrates how to measure and improve detection and response capabilities.
The article explains Microsoft’s Test‑In‑Production (TIP) approach, describing why production is the only true environment, how they use gradual releases, feature flags, telemetry, fault injection, circuit‑breaker testing, and chaos engineering to improve reliability, micro‑service compatibility, and business continuity.
The SRE Foundation course introduces site reliability engineering principles, practices, and tools, explaining why perfect reliability is impractical, outlining SRE responsibilities, detailing the curriculum across eight modules, and identifying the diverse professionals—from engineers to managers—who can benefit from mastering reliability, scalability, and automation.
The article explains how Home Depot (THD) adopted the VALET model—a five‑dimensional SLO language covering Volume, Availability, Latency, Error, and Ticket—to unify communication, automate data collection, and improve reliability across its massive retail and e‑commerce infrastructure.
This article uses everyday analogies to compare UDP and TCP, outlines their core characteristics, typical use cases, and describes how to add TCP-like reliability features to UDP such as connection establishment, ordering, loss recovery, flow control, and congestion control.
The article thoroughly explains Kafka’s message‑queue reliability design and implementation, covering use‑case scenarios, core concepts, storage format, producer acknowledgment settings, broker replication mechanisms (ISR, HW, LEO), consumer delivery semantics, the epoch solution for synchronization, and practical configuration guidelines for various consistency and availability requirements.
This article clarifies common misconceptions about Site Reliability Engineering, outlines the role’s responsibilities, presents the SRE Foundation course syllabus and target audience, and highlights the GOPS 2020 Global Operations Conference where the training is offered.
The article presents a comprehensive set of fifteen universal software architecture principles—ranging from redundancy and rollback to automation and non‑intrusive design—along with essential design guidelines such as separation of concerns, single responsibility, and low coupling to help architects build scalable, reliable, and maintainable systems.
This article explains RabbitMQ’s AMQP fundamentals, exchange types, reliability mechanisms such as confirms and returns, consumer flow control, idempotency, dead‑letter handling, and for various high‑availability deployment models including mirrored clusters and federation.
An in‑depth look at how JD Cloud’s high‑reliability storage architecture tackles data reliability challenges—covering replica management, redundancy, detection and repair mechanisms, tiered storage designs, and Intel Optane’s role in boosting performance—offering practical strategies for balancing cost and resilience.
The iQIYI FinTech team implemented a Chaos Engineering framework, using a purpose‑driven Chaos Monkey to inject controlled failures, validate high‑availability, isolation, and self‑healing of payment services, derive architectural improvements, build a fault‑case library, and transition from fault detection to proactive system robustness.
This article outlines ByteDance’s adoption of chaos engineering, describing its background, industry examples, the evolution of internal fault‑injection platforms across three generations, the fault model and center design, experiment principles, and future plans for infrastructure‑level chaos and automated diagnostics.
This article compares monolithic applications and microservice architectures across dimensions such as network latency, development and operational complexity, reliability, resource consumption, scaling precision, throughput, deployment speed, and team communication, highlighting where each approach wins and offering guidance on when to adopt microservices.
This article outlines the challenges of scaling parking services to cloud SaaS—including network latency, real‑time processing, and data integrity—and presents a comprehensive stability strategy using MQTT, dual‑network backup, Go‑based process supervision, and edge‑cloud collaboration to achieve high‑availability unattended parking operations.
This article explains Kafka's internal storage architecture—including topics, partitions, segments, .log and .index files—how data is read, and the various reliability mechanisms such as ISR/OSR, LEO/HW, producer acknowledgment levels, leader election strategies, and delivery semantics.