BestHub
Sign in
Tagged articles

fault injection

72 articles · Page 1 of 1
Data Bricklaying Diary
Data Bricklaying Diary
Sep 25, 2026 · Backend Development

Beyond Retry: Designing Fault Tolerance from Failure Models to Disaster Recovery

This article explains why retry alone is insufficient for fault tolerance, detailing how to classify failures via fault models, apply appropriate mechanisms like timeouts and circuit breakers, design recovery paths targeting state convergence, define RTO/RPO for disaster recovery, and validate all assumptions through fault injection drills.

Circuit BreakerRPORTO
0 likes · 23 min read
Beyond Retry: Designing Fault Tolerance from Failure Models to Disaster Recovery
Random Bulletin
Random Bulletin
Sep 7, 2026 · Operations

Fault Injection at Scale: Turning Paper Capacity into Trusted Resilience

This article explains why traditional capacity planning fails under real faults, detailing how to build continuous fault injection practices with hypothesis-driven experiments, blast radius control, automated stop conditions, and fault capacity maps to transform theoretical headroom into verified effective throughput at 10M QPS scale.

Capacity Planningblast radiuscapacity map
0 likes · 32 min read
Fault Injection at Scale: Turning Paper Capacity into Trusted Resilience
Machine Heart
Machine Heart
Jun 29, 2026 · Artificial Intelligence

Open‑Source AI‑Infra Ops Agent Benchmark Powered by Hundreds of Billions of Real Data

The article introduces AISHPerf, the first open‑source benchmark for AI‑infra operations agents built on nearly a hundred‑billion real‑world ops records, detailing its data pipeline, multi‑layer coverage, evaluation metrics, experimental results that show current models lag behind human experts, and future plans to expand and refine the benchmark.

AI OpsBenchmarkGPU cluster
0 likes · 16 min read
Open‑Source AI‑Infra Ops Agent Benchmark Powered by Hundreds of Billions of Real Data
Woodpecker Software Testing
Woodpecker Software Testing
Jun 8, 2026 · Industry Insights

Adversarial Testing in Practice: Building the Next Defense Line for Software Quality

The article examines how adversarial testing—beyond AI red‑team attacks—has become a system‑level quality safeguard, illustrating a real‑world financial app breach, a three‑layer testing framework, engineering maturity steps, and the evolving role of test engineers in resilient software development.

AI riskCI/CDadversarial testing
0 likes · 8 min read
Adversarial Testing in Practice: Building the Next Defense Line for Software Quality
Woodpecker Software Testing
Woodpecker Software Testing
Mar 2, 2026 · Industry Insights

Adversarial Testing in Practice: How It Outperforms Traditional Testing

The article explains how adversarial testing shifts from a user‑centric to an attacker‑centric paradigm, illustrates real‑world cases in finance, autonomous driving and AI, outlines perturbation layers, evaluation metrics, automation pipelines, and three counter‑intuitive principles for effective deployment, highlighting its advantages over conventional testing.

AI safetySoftware RobustnessTesting Methodology
0 likes · 8 min read
Adversarial Testing in Practice: How It Outperforms Traditional Testing
FunTester
FunTester
Jul 11, 2025 · Operations

Why Chaos Engineering Is Essential for Building Resilient Systems

This article explains how chaos engineering deliberately injects failures to reveal hidden weaknesses, helping organizations test and improve infrastructure resilience, handle traffic spikes, recover from disasters, and maintain continuous service in today’s always‑on digital environment.

Resilience Testingchaos engineeringfault injection
0 likes · 7 min read
Why Chaos Engineering Is Essential for Building Resilient Systems
FunTester
FunTester
May 19, 2025 · Operations

Chaos Engineering Tools, Theory, and Practices

Chaos engineering, a scientific method for improving system resilience, is explored through an overview of leading tools such as Gremlin, ChaosBlade, Chaos Mesh, Chaos Toolkit, and ChaosMeta, alongside core concepts, real-world case studies, common misconceptions, and the practical value of controlled fault injection in distributed systems.

chaos engineeringdistributed systemsfault injection
0 likes · 12 min read
Chaos Engineering Tools, Theory, and Practices
FunTester
FunTester
May 16, 2025 · Operations

Chaos Engineering: Evolution, Workflow, Advantages, and Practice Principles

Chaos engineering is a discipline that deliberately injects faults into distributed systems to test and improve resilience, tracing its evolution from Netflix's Chaos Monkey to modern platforms, outlining its operational workflow, benefits, and core principles for reliable system design.

SREchaos engineeringdistributed systems
0 likes · 9 min read
Chaos Engineering: Evolution, Workflow, Advantages, and Practice Principles
FunTester
FunTester
May 8, 2025 · Backend Development

Mastering HTTP Timeouts: Types, Causes, and Chaos Mesh Simulations

Understanding the three HTTP timeout types—connect, write, and read—helps engineers pinpoint failures, while detailed examples of causes and observable symptoms guide troubleshooting, and step-by-step Chaos Mesh simulations demonstrate how to inject and monitor these faults to validate system resilience.

BackendHTTPchaos engineering
0 likes · 17 min read
Mastering HTTP Timeouts: Types, Causes, and Chaos Mesh Simulations
FunTester
FunTester
Apr 21, 2025 · Backend Development

Sentinel: Flow Control and Circuit Breaking for Microservice Stability

This article explains how Sentinel, an open‑source flow‑control component from Alibaba, provides fine‑grained rate limiting, circuit breaking, and system protection for microservices, detailing its core mechanisms, configuration options, and practical usage in performance and fault testing.

Circuit BreakingFlow Controlfault injection
0 likes · 14 min read
Sentinel: Flow Control and Circuit Breaking for Microservice Stability
JD Tech
JD Tech
Apr 17, 2025 · Operations

Chaos Engineering: Principles, Core Steps, Tool Selection, and AI Integration

This article explains chaos engineering—its definition, core principles, experimental workflow, tool selection, AI‑driven enhancements, and practical case studies—providing a comprehensive guide for building resilient distributed systems across backend, cloud‑native, mobile, and AI‑enabled environments.

AI Integrationchaos engineeringdistributed systems
0 likes · 26 min read
Chaos Engineering: Principles, Core Steps, Tool Selection, and AI Integration
FunTester
FunTester
Mar 31, 2025 · Operations

Performance Testing and Fault Testing: Complementary Pillars for System Stability

The article explains how performance testing measures system efficiency under load while fault testing validates resilience under abnormal conditions, highlighting their shared goals, differences, overlapping toolchains, and how their combined use drives architecture optimization and improves service level agreements in modern complex software systems.

Load TestingResiliencefault injection
0 likes · 14 min read
Performance Testing and Fault Testing: Complementary Pillars for System Stability
Bilibili Tech
Bilibili Tech
Mar 18, 2025 · Operations

Technical Practices for Ensuring Stability of Bilibili’s 2025 Spring Festival Gala Live Stream

Bilibili’s engineering team built a scenario‑metadata and one‑click fault‑drill platform, implemented multi‑tier degradation, dynamic capacity planning, and extensive automated fault‑injection testing to guarantee zero‑severity incidents during the high‑traffic 2025 Spring Festival Gala live stream.

High Concurrencyfault injectionlive streaming
0 likes · 16 min read
Technical Practices for Ensuring Stability of Bilibili’s 2025 Spring Festival Gala Live Stream
FunTester
FunTester
Mar 12, 2025 · Operations

Fault Injection Testing: Concepts, Scenarios, Process, and Best Practices

Fault injection testing deliberately introduces failures into a system to assess its resilience, helping identify weak points, improve retry and timeout mechanisms, and ensure robust operation across software, protocol, and infrastructure layers, with practical guidance on processes, tools, and Kubernetes-specific practices.

Kuberneteschaos engineeringfault injection
0 likes · 8 min read
Fault Injection Testing: Concepts, Scenarios, Process, and Best Practices
FunTester
FunTester
Mar 5, 2025 · Backend Development

Calling Third‑Party Java Methods with Byteman in Chaos Mesh

This article demonstrates how to use Byteman’s DO execution module on the Chaos Mesh platform to invoke static or instance methods of external Java classes without modifying the original code, covering reflection, ClassLoader tricks, and a complete BTM rule example.

BytemanChaos MeshClassLoader
0 likes · 7 min read
Calling Third‑Party Java Methods with Byteman in Chaos Mesh
FunTester
FunTester
Feb 16, 2025 · Operations

Master Byteman: Install, Build, and Configure Java Fault Injection

This guide walks you through downloading Byteman, setting up BYTEMAN_HOME, using Ant or Maven for integration, building from source, configuring the Java agent with detailed options, and leveraging tutorials for effective fault‑injection testing in Java applications.

AntBytemanJava
0 likes · 8 min read
Master Byteman: Install, Build, and Configure Java Fault Injection
FunTester
FunTester
Jan 27, 2025 · Operations

Mastering Chaos Engineering: Build Resilient Systems with Proven Practices

In today's always‑on digital era, this article explains chaos engineering concepts, step‑by‑step experimental methods, best‑practice guidelines, and a comparison of leading fault‑injection tools to help organizations proactively strengthen system resilience and reduce downtime risk.

DevOpschaos engineeringcloud-native
0 likes · 11 min read
Mastering Chaos Engineering: Build Resilient Systems with Proven Practices
FunTester
FunTester
Jan 16, 2025 · Backend Development

Mastering Byteman: Injecting Bytecode for Advanced Java Testing

Byteman is a powerful Java bytecode manipulation tool that lets developers inject custom code at runtime without recompiling, using an event‑condition‑action rule language to trace, modify execution flow, coordinate threads, and collect statistics, with detailed examples of rule syntax, binding, and built‑in actions.

JavaRule Enginebytecode
0 likes · 12 min read
Mastering Byteman: Injecting Bytecode for Advanced Java Testing
FunTester
FunTester
Nov 4, 2024 · Backend Development

Mastering Java Fault Injection with Byteman: A Hands‑On Guide

Byteman is a dynamic Java fault‑injection tool that lets developers simulate network delays, service crashes, and resource exhaustion without altering source code, offering seamless integration with JUnit/TestNG, detailed rule definitions, and convenient shell scripts for installing, submitting, and removing fault‑injection rules.

BytemanJVMJava
0 likes · 12 min read
Mastering Java Fault Injection with Byteman: A Hands‑On Guide
DevOps
DevOps
Aug 22, 2024 · Operations

Synthetic Monitoring and Fault Drills: Practices for Ensuring Service Stability

This article explains why service stability is critical, outlines the importance and key factors of synthetic monitoring, provides practical guidelines for implementing it, and then describes fault‑drill concepts, benefits, processes, and common cloud‑native tools to proactively discover and mitigate failures in micro‑service environments.

Synthetic Monitoringfault injectionoperations
0 likes · 11 min read
Synthetic Monitoring and Fault Drills: Practices for Ensuring Service Stability
Aikesheng Open Source Community
Aikesheng Open Source Community
Jun 27, 2024 · Databases

Evaluation of OceanBase Arbitration Service in a 2F1A Deployment: Fault Injection Experiments and Recovery Procedures

This article presents a detailed experimental study of OceanBase's Arbitration Service in a 2F1A (two full‑function replicas plus one arbitration node) configuration, examining how the system behaves when one or both full‑function replicas fail, how log‑stream degradation and permanent offline mechanisms work, and how normal service is restored after node recovery.

Arbitration ServiceOceanBaseRecovery
0 likes · 17 min read
Evaluation of OceanBase Arbitration Service in a 2F1A Deployment: Fault Injection Experiments and Recovery Procedures
Efficient Ops
Efficient Ops
May 12, 2024 · Operations

From Firefighting to Fire‑Starting: Mastering Operations for System Reliability

The article outlines a three‑stage evolution of operations—from rapid incident response to proactive fault‑injection—while offering practical guidance on improving availability, visualizing changes, and aligning technical metrics with business value to elevate the role of operations engineers.

SREavailabilityfault injection
0 likes · 7 min read
From Firefighting to Fire‑Starting: Mastering Operations for System Reliability
FunTester
FunTester
Mar 29, 2024 · Operations

Implementing Chaos Engineering in WeChat Pay: Practices, Challenges, and Outcomes

This article describes how WeChat Pay applied chaos engineering to improve system reliability, detailing the business scenario, challenges of controlling fault injection radius, practical solutions, risk assessment, automation, and the resulting business and tool achievements.

WeChat Paychaos engineeringfault injection
0 likes · 18 min read
Implementing Chaos Engineering in WeChat Pay: Practices, Challenges, and Outcomes
High Availability Architecture
High Availability Architecture
Mar 21, 2024 · Operations

Applying Chaos Engineering to WeChat Pay: Design, Implementation, and Outcomes

To improve the ultra‑high availability of WeChat Pay, the team introduced chaos engineering using multi‑partition isolation, controlled blast radius, automated fault injection, and systematic risk discovery, detailing the design, execution, automation, and results of this reliability‑focused initiative.

WeChat Paychaos engineeringfault injection
0 likes · 18 min read
Applying Chaos Engineering to WeChat Pay: Design, Implementation, and Outcomes
Tencent Cloud Developer
Tencent Cloud Developer
Mar 19, 2024 · Operations

Chaos Engineering in WeChat Pay: Design, Implementation, and Results

WeChat Pay’s team adopted Netflix‑style chaos engineering, building an automated, YAML‑driven fault‑injection platform that isolates experiments in multi‑zone partitions, enabling over 500 safe experiments in 2021‑2022, uncovering critical bugs across core services while maintaining five‑nine availability and zero production incidents.

WeChat Payautomationchaos engineering
0 likes · 18 min read
Chaos Engineering in WeChat Pay: Design, Implementation, and Results
Meituan Technology Team
Meituan Technology Team
Feb 29, 2024 · Mobile Development

Meituan Technical Salon #77: Client‑Side Robustness Testing via Interface Data Mutation for Billion‑Traffic Systems

Meituan’s Technical Salon #77 presented a client‑side robustness testing framework that mutates API responses using semantic rules, injects them via a proxy, and detects crashes or hangs through static code scans and dynamic monitoring, employing array‑deduplication techniques to cut test volume while maintaining coverage, now deployed in Meituan and Youxuan apps.

Robustnessclient-side qualitydata mutation
0 likes · 15 min read
Meituan Technical Salon #77: Client‑Side Robustness Testing via Interface Data Mutation for Billion‑Traffic Systems
Bilibili Tech
Bilibili Tech
Nov 28, 2023 · Operations

Technical Assurance Practices for the 13th League of Legends World Championship Live Stream

For the 13th League of Legends World Championship live stream on Bilibili, a comprehensive technical‑assurance framework—covering pre‑event traffic buildup, in‑event experience, and post‑event replay—mapped over 60 business functions, applied a traffic‑estimation model, executed fault‑injection drills, load tests, strict SOPs and change control, and real‑time monitoring, enabling 120 million viewers and a peak of 460 million concurrent users.

Traffic Engineeringfault injectionlive streaming
0 likes · 19 min read
Technical Assurance Practices for the 13th League of Legends World Championship Live Stream
AntTech
AntTech
Nov 7, 2023 · Operations

ChaosMeta V0.6.0 Release: New Features, Lossless Injection, Automated Experiments, and Future Directions

ChaosMeta V0.6.0 introduces DNS and log injection capabilities, lossless fault injection concepts, automated experiment orchestration with atomic tasks, and a roadmap for multi‑cloud support and advanced metrics, aiming to solve the last‑mile challenge of continuous automated chaos experiments in production environments.

automated experimentschaos engineeringfault injection
0 likes · 9 min read
ChaosMeta V0.6.0 Release: New Features, Lossless Injection, Automated Experiments, and Future Directions
Huolala Tech
Huolala Tech
Aug 22, 2023 · Operations

How HuoLala Built a Resilient Fault‑Drill Platform to Boost System Reliability

Facing growing microservice complexity, HuoLala designed a comprehensive fault‑drill system—covering management, tooling, and operations—to simulate failures, control blast radius, automate scenarios, and continuously improve resilience, ultimately reducing downtime and enhancing system stability across more than ten business units.

automationchaos engineeringfault injection
0 likes · 12 min read
How HuoLala Built a Resilient Fault‑Drill Platform to Boost System Reliability
Bilibili Tech
Bilibili Tech
Aug 1, 2023 · Operations

Fault Injection Platform Design and Implementation Practice

The article details the design and deployment of a Java‑focused fault‑injection platform that uses Attach agents and JVM‑Sandbox to inject controllable pod‑ and request‑level faults—such as latency, exceptions, and return‑value errors—through dynamic templates, enabling fine‑grained, production‑safe chaos testing for e‑commerce services.

Java applicationJava developmentagent-based
0 likes · 15 min read
Fault Injection Platform Design and Implementation Practice
DevOps
DevOps
May 12, 2023 · Operations

Evolution of Chaos Engineering at Netflix: From Chaos Monkey to ChAP

This article examines how Netflix has progressively refined its chaos engineering practices—from the early Chaos Monkey tool to the sophisticated Chaos Automation Platform (ChAP)—to improve system resilience, automate experiments, and safely validate changes in large‑scale microservice environments.

Netflixchaos engineeringfault injection
0 likes · 26 min read
Evolution of Chaos Engineering at Netflix: From Chaos Monkey to ChAP
JD Tech
JD Tech
Mar 14, 2023 · Operations

Introduction to Chaos Engineering and Its Practical Exercise Workflow

This article offers a comprehensive overview of chaos engineering, explaining its definition, why it is needed, the value it brings, a detailed step‑by‑step practice workflow—including preparation, execution, recovery and review phases—typical drill scenarios, key assessment metrics, and risk‑control measures to improve system reliability and high‑availability.

chaos engineeringfault injectionrisk management
0 likes · 11 min read
Introduction to Chaos Engineering and Its Practical Exercise Workflow
FunTester
FunTester
Mar 13, 2023 · Operations

How Chaos Engineering Can Strengthen System Reliability: A Practical Guide

This article explains the origins and principles of chaos engineering, illustrates how fault‑injection scenarios expose system weaknesses, outlines step‑by‑step implementation—from tool selection and metric definition to execution and post‑mortem—and highlights its role in achieving high‑availability service level agreements.

DevOpschaos engineeringdistributed systems
0 likes · 10 min read
How Chaos Engineering Can Strengthen System Reliability: A Practical Guide
ByteDance SYS Tech
ByteDance SYS Tech
Feb 28, 2023 · Cloud Native

How ByteDance’s ARES Boosts Cloud‑Native Resilience with Chaos Engineering

This article explains ByteDance’s end‑to‑end chaos engineering practice for cloud‑native environments, covering its background, principles, comparison with traditional testing, the evolution of its internal platforms, and a detailed look at the Application Resilience Enhancement Service (ARES) and its core features.

KubernetesResiliencechaos engineering
0 likes · 17 min read
How ByteDance’s ARES Boosts Cloud‑Native Resilience with Chaos Engineering
Bilibili Tech
Bilibili Tech
Nov 18, 2022 · Operations

Chaos Engineering and Fault Injection System Design: Principles, Implementation, and Practice

Chaos Engineering and Fault Injection System Design combine steady-state hypotheses, controlled blast-radius experiments, and a lightweight interceptor layer using gRPC and protobuf to inject and report faults in micro-service architectures, enabling continuous testing, rapid MTTR reduction, and resilient services through automated, real-time experimentation and analysis.

Gofault injectiongRPC
0 likes · 15 min read
Chaos Engineering and Fault Injection System Design: Principles, Implementation, and Practice
Architects Research Society
Architects Research Society
Sep 11, 2022 · Cloud Native

Chaos Mesh: A Cloud‑Native Chaos Engineering Platform for Kubernetes

Chaos Mesh, a CNCF‑hosted cloud‑native chaos engineering platform, orchestrates fault injection experiments in Kubernetes through components like the Chaos Operator and Dashboard, supporting various CRD types such as DNSChaos, PodChaos, and NetworkChaos to simulate failures ranging from pod kills to network partitions.

Chaos Meshfault injectionreliability testing
0 likes · 7 min read
Chaos Mesh: A Cloud‑Native Chaos Engineering Platform for Kubernetes
FunTester
FunTester
Jul 24, 2022 · Operations

Boost Service Reliability with Chaos Engineering: Practical Steps & Evaluation

Chaos engineering, a discipline for experimenting on distributed systems, helps teams identify hidden weaknesses, improve high‑availability, and build confidence in production by defining stable states, injecting realistic failures, and measuring impact through observability metrics, with practical steps, tool choices, maturity stages, and evaluation methods.

chaos engineeringdistributed systemsfault injection
0 likes · 11 min read
Boost Service Reliability with Chaos Engineering: Practical Steps & Evaluation
ITPUB
ITPUB
Jun 27, 2022 · Big Data

How Kuaishou Guarantees Real‑Time Data Warehouse Performance at Billion‑Scale Events

This article details Kuaishou's real‑time data warehouse architecture, the business challenges of massive traffic and diverse requirements, and the forward‑ and reverse‑assurance strategies—including lifecycle standards, monitoring, fault‑injection testing, and a Spring Festival case study—that together ensure high stability, low latency, and sub‑0.5% accuracy for billion‑scale streaming workloads.

Flink streamingKuaishouLarge‑scale data processing
0 likes · 22 min read
How Kuaishou Guarantees Real‑Time Data Warehouse Performance at Billion‑Scale Events
Qunar Tech Salon
Qunar Tech Salon
Jun 16, 2022 · Operations

Practical Chaos Engineering Practices at Qunar Travel: Architecture, Scenarios, and Automation

This article details Qunar Travel's mature chaos engineering platform built on chaosblade, covering value analysis, system architecture, shutdown and dependency drills, automated closed‑loop testing, attack‑defense exercises, and the measurable reliability improvements achieved across thousands of services.

chaos engineeringdistributed systemsfault injection
0 likes · 18 min read
Practical Chaos Engineering Practices at Qunar Travel: Architecture, Scenarios, and Automation
Bilibili Tech
Bilibili Tech
May 13, 2022 · Cloud Native

Chaos Engineering Practices for Bilibili Distributed KV Storage

Peng Liangyou describes how Bilibili’s large‑scale distributed KV storage adopts Netflix‑style chaos engineering—defining steady‑state hypotheses, replicating production environments, injecting CPU, memory, network and replica faults via automated “monkey” experiments, monitoring latency and durability with Prometheus/Grafana, and over 1.5 years preventing critical incidents while cutting testing costs and enabling incremental, standards‑based reliability improvements.

BilibiliKV storechaos engineering
0 likes · 15 min read
Chaos Engineering Practices for Bilibili Distributed KV Storage
AntTech
AntTech
Jan 24, 2022 · Operations

Ant Group's Chaos Engineering System: Evolution, Business Features, Key Technologies, and Future Directions

This article outlines Ant Group's six‑year journey in chaos engineering, describing its three generational evolutions, business‑oriented fault injection, risk‑mining, full‑lifecycle coverage, massive scale, root‑data protection, core technologies such as Awatch, simulation environments, and plans for intelligent, open‑source future development.

Ant GroupAwatchRisk Mining
0 likes · 23 min read
Ant Group's Chaos Engineering System: Evolution, Business Features, Key Technologies, and Future Directions
Alibaba Cloud Native
Alibaba Cloud Native
Dec 30, 2021 · Operations

How to Implement Chaos Engineering for Cloud‑Native Applications: A Step‑by‑Step Guide

This article explains how cloud‑native teams can adopt chaos engineering—defining its concepts, outlining its unique characteristics, and detailing a four‑stage implementation process from manual drills to production‑level raids, with practical steps, environment setups, and real‑world results.

Kuberneteschaos engineeringcloud-native
0 likes · 14 min read
How to Implement Chaos Engineering for Cloud‑Native Applications: A Step‑by‑Step Guide
GrowingIO Tech Team
GrowingIO Tech Team
Dec 2, 2021 · Cloud Native

Mastering Chaos Mesh: A Hands‑On Guide to Cloud‑Native Chaos Engineering

Chaos Mesh is an open‑source cloud‑native chaos engineering platform that lets you experiment with fault injection across Kubernetes environments, offering visual dashboards, extensive fault types, and step‑by‑step installation and experiment creation guides to help teams uncover system weaknesses and improve resilience.

Chaos MeshKuberneteschaos engineering
0 likes · 12 min read
Mastering Chaos Mesh: A Hands‑On Guide to Cloud‑Native Chaos Engineering
Alibaba Cloud Native
Alibaba Cloud Native
Aug 6, 2021 · Operations

Scaling Chaos Engineering at Qunar: Lessons from Thousands of Microservices

Qunar shares how it built a large‑scale chaos engineering platform for thousands of microservices, detailing tool selection, architecture, evolution stages, fault‑injection scenarios, strong/weak dependency automation, open‑source contributions, and future plans for automated random drills.

chaos engineeringcloud-nativefault injection
0 likes · 9 min read
Scaling Chaos Engineering at Qunar: Lessons from Thousands of Microservices
HelloTech
HelloTech
Jul 30, 2021 · Operations

Foundations of High Availability: Defining and Managing Strong and Weak Service Dependencies

The article defines strong versus weak service dependencies, outlines governance through discovery, fault injection, and refactoring, recommends front‑end and back‑end fault‑tolerance measures such as timeouts and circuit breakers, describes isolation and artificial degradation switches, verifies classifications, and notes current middleware gaps and hiring information.

Backendfault injectionfrontend
0 likes · 10 min read
Foundations of High Availability: Defining and Managing Strong and Weak Service Dependencies
iQIYI Technical Product Team
iQIYI Technical Product Team
Jul 2, 2021 · Cloud Native

Chaos Engineering Practices at iQIYI: Building Resilient Cloud‑Native Systems

iQIYI’s Little Deer Chaos Platform injects faults and runs red‑blue attacks across production services, enabling teams to validate alerts, circuit‑breakers, and fail‑over mechanisms—demonstrated by video playback and membership service case studies—thereby fostering zero‑trust design, faster skill growth, and resilient cloud‑native operations.

DevOpsResilience Testingchaos engineering
0 likes · 10 min read
Chaos Engineering Practices at iQIYI: Building Resilient Cloud‑Native Systems
DevOps
DevOps
Jun 2, 2021 · Operations

Disasterpiece Theater: Slack’s Process for Approachable Chaos Engineering

Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.

Slackchaos engineeringfault injection
0 likes · 11 min read
Disasterpiece Theater: Slack’s Process for Approachable Chaos Engineering
Alibaba Terminal Technology
Alibaba Terminal Technology
Jan 7, 2021 · Frontend Development

How Frontend Chaos Engineering Boosts Reliability: Lessons from Alibaba

This article explores the challenges of frontend availability, introduces chaos engineering concepts, and details Alibaba's practical approach to frontend fault injection—including static resource hijacking, a safe isolated environment, monitoring integration, and a real‑world drill that demonstrates how to measure and improve detection and response capabilities.

chaos engineeringfault injectionfrontend
0 likes · 18 min read
How Frontend Chaos Engineering Boosts Reliability: Lessons from Alibaba
Pupu Technology
Pupu Technology
Oct 12, 2020 · Operations

From Network Jitter to Fault Drills: A Real‑World Redisson Bug Investigation

The article recounts a production outage caused by cloud‑platform network jitter, details step‑by‑step fault diagnosis using logs, APM and monitoring, reproduces the issue with chaos‑engineered Redis/Redisson faults, and outlines how the team turned the incident into a systematic fault‑drill practice.

Redischaos engineeringfault injection
0 likes · 20 min read
From Network Jitter to Fault Drills: A Real‑World Redisson Bug Investigation
Efficient Ops
Efficient Ops
Sep 8, 2020 · Operations

From Firefighting to Arson: Mastering Ops Availability in Three Stages

The article outlines a three‑stage ops maturity model—firefighting, fire prevention, and arson—explains how proactive fault‑injection drills, continuous availability improvements, and aligning technical metrics with business value can transform operations from reactive responders into strategic value creators.

availabilityfault injectionincident management
0 likes · 8 min read
From Firefighting to Arson: Mastering Ops Availability in Three Stages
DevOps
DevOps
Aug 13, 2020 · Operations

ByteDance’s Chaos Engineering Journey: Practices, Architecture, and Future Directions

This article outlines ByteDance’s adoption of chaos engineering, describing its background, industry examples, the evolution of internal fault‑injection platforms across three generations, the fault model and center design, experiment principles, and future plans for infrastructure‑level chaos and automated diagnostics.

chaos engineeringdistributed systemsfault injection
0 likes · 21 min read
ByteDance’s Chaos Engineering Journey: Practices, Architecture, and Future Directions
Full-Stack Internet Architecture
Full-Stack Internet Architecture
Jul 22, 2020 · Operations

Building a Comprehensive High‑Availability System: Disaster Recovery, Capacity Planning, Online Protection, and Fault Drills

This article explains how to construct a truly high‑availability architecture for modern distributed, cloud‑native services by covering disaster‑recovery principles, capacity planning with realistic load testing, online traffic protection, and systematic fault‑drill practices.

Capacity Planningdisaster recoveryfault injection
0 likes · 13 min read
Building a Comprehensive High‑Availability System: Disaster Recovery, Capacity Planning, Online Protection, and Fault Drills
DataFunTalk
DataFunTalk
Apr 27, 2020 · Operations

ByteDance’s Chaos Engineering Practice and Platform Evolution

This article describes ByteDance’s multi‑generation chaos engineering practice, covering industry background, fault‑injection models, the design of a declarative fault‑center, experiment selection principles, detailed experiment processes, metric classifications, red‑blue war‑game workflows, strong/weak dependency analysis, and future directions for infrastructure‑level chaos engineering.

Platform Designchaos engineeringfault injection
0 likes · 21 min read
ByteDance’s Chaos Engineering Practice and Platform Evolution
Ctrip Technology
Ctrip Technology
Nov 14, 2019 · Operations

Chaos Engineering: Principles, Practices, and Lessons from Ctrip

The article explains Chaos Engineering as a discipline for deliberately injecting failures into distributed systems to uncover hidden weaknesses, outlines its five core principles, describes practical implementation steps and real‑world examples from Ctrip, and discusses future directions for reliability engineering.

chaos engineeringdistributed systemsfault injection
0 likes · 9 min read
Chaos Engineering: Principles, Practices, and Lessons from Ctrip
Programmer DD
Programmer DD
Aug 4, 2019 · Operations

Simulating CPU and I/O Failures with Bash Scripts for Chaos Engineering

This article demonstrates how to create Bash scripts that fully saturate CPU and I/O resources, explains their role in fault injection within the Simian Army framework, and introduces the broader concepts and benefits of chaos engineering for building resilient distributed systems.

bash scriptschaos engineeringcpu stress
0 likes · 9 min read
Simulating CPU and I/O Failures with Bash Scripts for Chaos Engineering
High Availability Architecture
High Availability Architecture
Jul 5, 2019 · Operations

Practices of Chaos Engineering in Distributed Service Architecture

This article presents a comprehensive overview of chaos engineering, covering its definition, value, principles, implementation steps, enterprise adoption strategies, the open‑source ChaosBlade tool and AHAS Chaos platform, and two detailed case studies demonstrating fault injection experiments in a distributed service environment.

AHASAlibabachaos engineering
0 likes · 15 min read
Practices of Chaos Engineering in Distributed Service Architecture
Alibaba Cloud Developer
Alibaba Cloud Developer
Jul 5, 2019 · Backend Development

How JVM‑Sandbox Boosts Alibaba’s Double‑11 Stability with Real‑Time Bytecode Enhancement

JVM‑Sandbox, an open‑source real‑time, non‑intrusive bytecode‑enhancement framework developed by Alibaba’s Technical Quality team since 2016, provides dynamic AOP, modular management, and HTTP‑based control to support fault injection, dependency analysis, recording/replay, and precise regression, dramatically improving testing efficiency and stability for large‑scale services.

Bytecode Instrumentationfault injectionjvm-sandbox
0 likes · 9 min read
How JVM‑Sandbox Boosts Alibaba’s Double‑11 Stability with Real‑Time Bytecode Enhancement
Alibaba Cloud Native
Alibaba Cloud Native
Mar 12, 2019 · Cloud Native

Injecting Real-World Failures into Ali Kubernetes with Open‑Source Monkey Tools

This article explains how chaos engineering principles are applied to Ali Kubernetes by reviewing open‑source Kubernetes monkey tools, analyzing complex failure scenarios, and presenting a custom fault‑injection suite built on the internal MonkeyKing platform to enable flexible, scenario‑driven chaos experiments.

KubernetesMonkey Toolschaos engineering
0 likes · 10 min read
Injecting Real-World Failures into Ali Kubernetes with Open‑Source Monkey Tools
Java Backend Technology
Java Backend Technology
Mar 2, 2019 · Operations

How Alibaba’s ‘MonkeyKing’ Uses Chaos Engineering to Strengthen System Reliability

Alibaba’s MonkeyKing, inspired by Netflix’s Chaos Monkey, employs intentional fault injection—from random node kills to simulated network outages—to test and improve system robustness across IaaS, PaaS, and SaaS layers, offering a comprehensive model for reliability engineering in complex distributed environments.

AlibabaNetflixchaos engineering
0 likes · 8 min read
How Alibaba’s ‘MonkeyKing’ Uses Chaos Engineering to Strengthen System Reliability
AntTech
AntTech
Dec 19, 2018 · Information Security

Red‑Blue Technical Attack‑Defense Exercises and SRE Practices at Ant Financial

Ant Financial’s internal red‑blue technical attack‑defense program, driven by a dedicated blue team and SRE‑based red team, continuously probes system weaknesses, refines fault‑injection tools like Awatch, and evolves high‑availability and self‑healing mechanisms to strengthen risk control and operational reliability.

SREfault injectioninformation security
0 likes · 10 min read
Red‑Blue Technical Attack‑Defense Exercises and SRE Practices at Ant Financial
Meituan Technology Team
Meituan Technology Team
Dec 13, 2018 · Operations

Stability Testing Practices for Meituan Smart Payment: Fault Drills, Online Load Testing, and Continuous Operation

Meituan’s smart‑payment team combats growing complexity and third‑party failures by implementing a stability‑building program that raises availability through flexible degradation, rapid recovery, and three core QA practices—fault drills, online full‑link load testing, and a continuous operation system that standardizes processes, visualizes metrics, and automates resilience checks.

Load TestingMeituanQA
0 likes · 13 min read
Stability Testing Practices for Meituan Smart Payment: Fault Drills, Online Load Testing, and Continuous Operation
dbaplus Community
dbaplus Community
Sep 11, 2018 · Operations

How Qunar’s Fault Injection Platform Ensures High‑Availability in Complex Backend Systems

Qunar built a fault‑injection platform that dynamically injects runtime errors into its densely coupled backend services, enabling verification of degradation and circuit‑breaker strategies, with a four‑part architecture comprising a web UI, deployment system, command server, and Java agents using Instrumentation‑API for bytecode weaving.

AOPBackendJava Instrumentation
0 likes · 13 min read
How Qunar’s Fault Injection Platform Ensures High‑Availability in Complex Backend Systems
Qunar Tech Salon
Qunar Tech Salon
Aug 8, 2018 · Backend Development

Design and Implementation of a Fault Injection Platform for High‑Availability Backend Systems

This article describes the motivation, architecture, and implementation details of a fault‑injection platform that uses Java Instrumentation and dynamic bytecode weaving to validate high‑availability strategies, isolate failures, and support zero‑cost, runtime fault injection for complex distributed backend services.

AOPBackendJava Instrumentation
0 likes · 12 min read
Design and Implementation of a Fault Injection Platform for High‑Availability Backend Systems
Youzan Coder
Youzan Coder
Jun 22, 2018 · Operations

Chaos Engineering: Definition, Principles, and Implementation Steps

Chaos engineering is a disciplined practice that injects controlled faults into distributed systems—often in production—to validate steady-state hypotheses, uncover hidden reliability weaknesses, and continuously improve resilience, as illustrated by the staged implementations and fault-injection techniques used by companies such as JD.com, Youzan, and Netflix.

chaos engineeringfault injectionreliability
0 likes · 11 min read
Chaos Engineering: Definition, Principles, and Implementation Steps
ITPUB
ITPUB
Nov 11, 2017 · Backend Development

How JD.com Scaled Double‑11 with Dynamic Load Balancing, Rate Limiting, and AI‑Driven Upgrades

This article examines JD.com’s technical strategies for the 2023 Double‑11 shopping festival, detailing dynamic load‑balancing and rate‑limiting mechanisms, evolving fault‑drill practices, and AI‑powered product and marketing enhancements that together ensure high‑concurrency stability and improved user experience.

AIJD.comRate Limiting
0 likes · 14 min read
How JD.com Scaled Double‑11 with Dynamic Load Balancing, Rate Limiting, and AI‑Driven Upgrades
Meituan Technology Team
Meituan Technology Team
Jun 23, 2017 · Backend Development

Fault Drill: Traffic Replication and Fault Injection Platform for Hotel Backend

The Fault‑Drill platform for hotel back‑end services combines real‑time traffic replication to shadow clusters with UI‑driven fault injection via a java‑agent, enabling developers to validate incident‑response plans, measure latency impacts, and reduce MTTR by testing normal and abnormal conditions on live traffic.

backend engineeringdistributed systemsfault injection
0 likes · 13 min read
Fault Drill: Traffic Replication and Fault Injection Platform for Hotel Backend