This article breaks down Kubernetes troubleshooting into three essential steps—understanding the failure, managing the response, and preventing recurrence—while mapping key monitoring, observability, and incident‑response tools to each phase for reliable cloud‑native operations.
The DeWu Tech Salon, co‑hosted by DeWu App Quality Platform and TesterHome, brought senior engineers from Alibaba Cloud, ByteDance, Lagou and DeWu together to share practical QA insights on end‑side monitoring, traffic replay, full‑link stress testing, and industry‑scale chaos engineering, while announcing a PPT collection, a testing‑expert recruitment drive, and a preview of the next wireless‑technology salon.
This article explores Qunar's practical implementation of chaos engineering, detailing its value, the four strategic directions, shutdown and application drills, strong‑weak dependency handling, container support, and automated closed‑loop testing that together boost system resilience, process robustness, and user experience.
This article outlines Ant Group's six‑year journey in chaos engineering, describing its three generational evolutions, business‑oriented fault injection, risk‑mining, full‑lifecycle coverage, massive scale, root‑data protection, core technologies such as Awatch, simulation environments, and plans for intelligent, open‑source future development.
This guide walks through a hybrid cloud disaster‑recovery demo for an e‑commerce platform, detailing business background, requirements, challenges, the MSHA‑based active‑active solution, step‑by‑step implementation, and verification of sub‑minute RPO/RTO using Alibaba Cloud services and Chaos engineering.
This article explains how cloud‑native teams can adopt chaos engineering—defining its concepts, outlining its unique characteristics, and detailing a four‑stage implementation process from manual drills to production‑level raids, with practical steps, environment setups, and real‑world results.
Based on 1,016 valid questionnaire responses and 17 enterprise interviews, the 2021 China Chaos Engineering Survey Report reveals low software system stability, limited adoption of chaos engineering, its positive impact on availability, and provides data‑driven recommendations for improving stability through mature tools, metrics, and cultural shifts.
This article examines the stability challenges of selecting distributed data products, introduces chaos‑engineering‑based testing methods, outlines practical test scenarios, fault injection techniques, toolchains, and quantitative analysis metrics, and presents a capability assessment standard for ensuring system reliability.
Chaos Mesh is an open‑source cloud‑native chaos engineering platform that lets you experiment with fault injection across Kubernetes environments, offering visual dashboards, extensive fault types, and step‑by‑step installation and experiment creation guides to help teams uncover system weaknesses and improve resilience.
This article describes how to model a complex Uber‑style ride‑hailing system using Domain‑Driven Design, implement it with Java Spring Boot microservices, instrument it with OpenTelemetry for full observability, and validate the observability pipeline through a gamified chaos‑engineering approach that reduces MTTR.
This article introduces a collection of Alibaba‑released open‑source tools—including Arthas, Cloud Toolkit, ChaosBlade, PTS, Druid, and more—detailing their usage scenarios, tutorials, and acquisition methods to help backend developers improve efficiency, debugging, monitoring, and reliability of their services.
This article explains ByteDance's journey from basic fault‑injection testing to a production‑grade, scenario‑driven proactive chaos engineering platform that automates experiments, defines stability metrics, controls blast radius, and continuously validates service dependencies to improve system resilience.
This article explains the design, model, and practical usage of Alibaba's open‑source ChaosBlade and its platform chaosblade‑box, detailing how a unified chaos experiment model enables scalable, multi‑environment fault injection for cloud‑native systems and improves high‑availability testing.
This article explains how to deploy the ChaosBlade Operator on a KubeSphere cluster, define and run various chaos experiments—such as CPU overload, network latency, packet loss, and process kill/stop—against RadonDB MySQL containers, and verify the impact to improve system resilience and fault‑tolerance.
This article explains that chaos engineering is not a magic cure but a disciplined practice for testing distributed systems by designing and running controlled experiments, outlining four essential steps—observability, defining steady state, hypothesizing events, and executing experiments—to gain confidence in system resilience.
Qunar shares how it built a large‑scale chaos engineering platform for thousands of microservices, detailing tool selection, architecture, evolution stages, fault‑injection scenarios, strong/weak dependency automation, open‑source contributions, and future plans for automated random drills.
This article explains how to use chaos engineering with Gremlin to design, execute, and analyze four experiments that test Kafka broker load, message loss, split‑brain scenarios, and ZooKeeper outages, helping improve the reliability and resilience of Kafka deployments.
The article introduces Security Chaos Engineering (SCE) as a proactive experimental approach to uncover security control failures, discusses the limitations of traditional red/blue/purple team exercises, outlines SCE's advantages, and presents the open‑source ChaoSlingr framework as a practical implementation example.
iQIYI’s Little Deer Chaos Platform injects faults and runs red‑blue attacks across production services, enabling teams to validate alerts, circuit‑breakers, and fail‑over mechanisms—demonstrated by video playback and membership service case studies—thereby fostering zero‑trust design, faster skill growth, and resilient cloud‑native operations.
Netflix describes how it extended the academic Molly fault‑injection method into an automated, production‑scale failure‑testing system that explores dependency trees, defines success criteria, runs short low‑impact experiments, and discovers hidden faults before they affect users.
Slack’s resilience engineering team outlines a structured chaos‑engineering workflow—identifying potential failures, ensuring fault tolerance, and deliberately injecting faults in development and production—to safely test system robustness, validate hypotheses, and continuously improve reliability through regular disaster‑theater exercises.
This article details how a major securities firm analyzed business stability, built a comprehensive stability engineering platform using chaos engineering, practiced extensive fault‑injection drills, and outlines future directions such as random‑scenario exercises, red‑blue battles, and AI‑driven risk detection.
Amid accelerating Agile and DevOps adoption, the rapid delivery of cloud‑native microservices introduces cascading risks and makes traditional monolithic testing inadequate, prompting a shift toward observability‑driven “right‑shift” testing, exploratory methods, and chaos engineering to embrace failure as the new normal.
The article explains how traditional testing falls short for cloud‑native, microservice‑based applications and outlines modern strategies—including unit, integration, contract, non‑functional, chaos engineering, and observability techniques—to ensure quality, resilience, and rapid delivery in dynamic cloud environments.
ChaosBlade, Alibaba’s open‑source chaos engineering project now advancing through CNCF Sandbox, offers a comprehensive suite—including the chaosblade experiment tool and chaosblade‑box platform—to simulate over 200 scenarios across hosts, Kubernetes, and multi‑language applications, with automated deployment, extensible architecture, and enterprise adoption examples.
This article presents a curated list of popular open‑source tools for monitoring, deployment, chaos testing, and reliability engineering, explaining their main features and how they help SREs and DevOps engineers build scalable, highly available cloud‑native systems.
The article introduces ChaosBlade‑Box, an open‑source cloud‑native chaos‑engineering console that builds on Alibaba’s ChaosBlade tool, explains the high‑availability challenges of cloud‑native systems, details the platform’s design, features, multi‑language support, deployment workflow, example experiments, and future roadmap for resilient architectures.
Baidu Testing Middleware is an Envoy‑based sidecar proxy that combines a data‑plane and control‑plane to intercept, inspect, modify, and route traffic, providing recording, replay, fault injection and rate‑limiting capabilities that support functional, system, integration, sandbox and chaos testing at massive scale.
The article introduces Baidu's self‑developed test middleware, explains its data‑plane and control‑plane architecture inspired by Istio, describes the challenges of building test environments, and details the system’s components, working modes, and diverse testing use cases such as functional, system, integration, sandbox, and chaos engineering.
This article explores the challenges of frontend availability, introduces chaos engineering concepts, and details Alibaba's practical approach to frontend fault injection—including static resource hijacking, a safe isolated environment, monitoring integration, and a real‑world drill that demonstrates how to measure and improve detection and response capabilities.
Tencent Cloud’s network operations platform combines a layered underlay‑overlay architecture, rapid fault detection within seconds and recovery in minutes, chaos‑engineering experiments, rigorous change health checks, high‑frequency multi‑path monitoring, and plans for predictive self‑healing to ensure reliable service across millions of servers.
This article explains the challenges of achieving high availability in unreliable environments, introduces disaster‑tolerance concepts and RPO/RTO metrics, describes Alibaba Cloud's AHAS‑MSHA multi‑site solution and its key features, and walks through two e‑commerce case studies that demonstrate implementation steps, fault‑injection drills, and recovery verification.
The article explains how Netflix uses Chaos Monkey and a suite of related tools, collectively called the Simian Army, to deliberately inject failures into their cloud infrastructure, continuously test fault‑tolerance, and ensure high availability and reliability for their streaming service.
The article recounts a production outage caused by cloud‑platform network jitter, details step‑by‑step fault diagnosis using logs, APM and monitoring, reproduces the issue with chaos‑engineered Redis/Redisson faults, and outlines how the team turned the incident into a systematic fault‑drill practice.
This article explains the need for chaos engineering in modern distributed and cloud‑native systems, outlines the challenges faced by architects, developers, testers and product teams, and provides step‑by‑step guidance on using ChaosBlade and Alibaba's AHAS platform for effective fault‑injection experiments.
The iQIYI FinTech team implemented a Chaos Engineering framework, using a purpose‑driven Chaos Monkey to inject controlled failures, validate high‑availability, isolation, and self‑healing of payment services, derive architectural improvements, build a fault‑case library, and transition from fault detection to proactive system robustness.
This article explains the fundamentals of cloud native computing, introduces service mesh architectures such as Istio and Envoy, explores chaos engineering with Chaos Mesh, and details how eBPF/XDP‑based user‑space container networking can accelerate data‑plane performance in modern microservice environments.
This article outlines ByteDance’s adoption of chaos engineering, describing its background, industry examples, the evolution of internal fault‑injection platforms across three generations, the fault model and center design, experiment principles, and future plans for infrastructure‑level chaos and automated diagnostics.
Site Reliability Engineeringchaos engineeringdistributed systems
This article details KuJiaLe's journey from monolithic to micro‑service architecture, the stability challenges encountered, and how they designed and deployed a ChaosBlade‑based fault‑injection platform that improves fault tolerance, accelerates incident response, and enhances overall user experience.
This article describes Ctrip's SRE team's journey in adopting chaos engineering, outlining the motivations, roadmap, concrete experiments, platform maturity, and future automation goals to improve system resilience and operational reliability in a large‑scale microservice environment.
This article explains chaos engineering as a systematic approach to experiment on distributed systems, identifies common failure modes, outlines a four‑step experimentation process, and presents advanced principles to help teams increase reliability and confidence in production environments.
This article explains why online applications experience crashes during traffic spikes, outlines the complexity of modern cloud‑based service architectures, and shares Alibaba engineers’ practical notes on high‑availability design, capacity planning, full‑link stress testing, monitoring, traffic control, routine inspections, and chaos‑engineering drills using tools such as AHAS, PTS, Sentinel and Advisor.
The article explains Chaos Engineering as a discipline for deliberately injecting failures into distributed systems to uncover hidden weaknesses, outlines its five core principles, describes practical implementation steps and real‑world examples from Ctrip, and discusses future directions for reliability engineering.
This article introduces Alibaba's open‑source ChaosBlade tool, explains its experiment model and supported scenarios, shows how to install the ChaosBlade Operator on Kubernetes, and provides step‑by‑step instructions for creating, modifying, and cleaning up cloud‑native chaos experiments using both YAML resources and the blade CLI.
This article presents a comprehensive overview of Netflix's chaos engineering journey, detailing its origins, the development of the Simian Army tools, core principles, practical steps, and applications in Kubernetes environments, offering valuable insights for reliable DevOps practices.
This article demonstrates how to create Bash scripts that fully saturate CPU and I/O resources, explains their role in fault injection within the Simian Army framework, and introduces the broader concepts and benefits of chaos engineering for building resilient distributed systems.
This article presents a comprehensive overview of chaos engineering, covering its definition, value, principles, implementation steps, enterprise adoption strategies, the open‑source ChaosBlade tool and AHAS Chaos platform, and two detailed case studies demonstrating fault injection experiments in a distributed service environment.
YouZan’s middleware testing team, divided into six specialized groups, employs a left‑shift quality assurance system spanning requirements, development, testing, release, and go‑live phases—leveraging over 10,000 automated test cases, a bus‑based release framework, chaos engineering, continuous delivery, and comprehensive monitoring tools to ensure resilient, high‑quality services.
This article introduces a curated selection of Alibaba’s open‑source and cloud‑based developer tools—including Arthas, Cloud Toolkit, ChaosBlade, ARMS, Docsite, Freeline, EasyExcel, Druid, Dragonwell and more—detailing their use cases, tutorials, and acquisition methods to help developers improve efficiency and code quality.
ChaosBlade is an open‑source chaos engineering tool from Alibaba that lets you repeatedly inject failures into distributed systems, helping you measure fault tolerance, validate orchestration, test platform robustness, verify monitoring alerts, and improve emergency response capabilities for more reliable cloud‑native applications.
This article explains how chaos engineering principles are applied to Ali Kubernetes by reviewing open‑source Kubernetes monkey tools, analyzing complex failure scenarios, and presenting a custom fault‑injection suite built on the internal MonkeyKing platform to enable flexible, scenario‑driven chaos experiments.
Alibaba’s MonkeyKing, inspired by Netflix’s Chaos Monkey, employs intentional fault injection—from random node kills to simulated network outages—to test and improve system robustness across IaaS, PaaS, and SaaS layers, offering a comprehensive model for reliability engineering in complex distributed environments.
This article explains why chaos engineering is essential for distributed systems, distinguishes it from traditional fault testing, outlines practical inputs and principles, and details Alibaba's multi‑year experience with fault‑drill platforms, automation, and future plans to improve system reliability.
Chaos engineering is a disciplined practice that injects controlled faults into distributed systems—often in production—to validate steady-state hypotheses, uncover hidden reliability weaknesses, and continuously improve resilience, as illustrated by the staged implementations and fault-injection techniques used by companies such as JD.com, Youzan, and Netflix.
The article explores how anti‑fragility principles—drawn from Nassim Taleb’s theory, Jez Humble’s training, Phoenix Server, Netflix’s Chaos Monkey, Amazon GameDays, and Etsy’s blameless post‑mortems—can be applied to software engineering to turn system failures into opportunities for growth and stronger organizational culture.
This article explores anti‑fragility theory and real‑world DevOps practices such as Phoenix Server, Chaos Monkey, GameDays, and blameless post‑mortems, showing how organizations can transform inevitable failures into opportunities for resilience and continuous improvement.
This article presents a comprehensive overview of fault governance in large‑scale distributed systems, covering classic dependency failures, the concept of strong and weak dependencies, experimental observations, the evolution of fault‑injection techniques, and best practices for building reliable fault‑drill platforms.
This article recounts Alibaba's middleware team's QCon Beijing 2017 presentation on fault governance and fault‑drill practices, covering distributed‑system dependency failures, strong/weak dependency concepts, multi‑stage technical evolution, and the design of their chaos‑engineering platform for large‑scale e‑commerce.
This article walks through designing a microservice application with Go, covering service registration and discovery, various testing strategies—including unit, integration, contract, and chaos testing—illustrates typical architectural patterns, and shares Qiniu's real‑world microservice implementation and practical Q&A.