Practices of Chaos Engineering in Distributed Service Architecture

Speaker Xiao Changjun (nickname Qionggu), a senior development engineer from Alibaba's High‑Availability Architecture team, introduces his background in distributed systems design, APM development, and his role as the lead of the open‑source ChaosBlade project and core developer of Alibaba Cloud AHAS.

Chaos engineering is defined as the discipline of conducting experiments on distributed systems to improve fault tolerance and recoverability. Its value includes validating architectural resilience, enhancing incident response efficiency, filling gaps left by traditional testing, and improving user experience through systematic fault injection.

The five guiding principles are: (1) hypothesize stable‑state behavior using business‑level metrics, (2) simulate realistic faults, (3) run experiments in production‑like environments, (4) automate continuous execution, and (5) control the blast radius to limit unintended impact.

Implementation follows eight steps: define experiment plan, specify steady‑state metrics, formulate fault‑tolerance hypotheses, execute the experiment, verify metrics, record and recover, fix discovered issues, and continuously validate.

Enterprise adoption is described in three phases: firmly articulate the value of chaos engineering, introduce the necessary tooling, and promote a culture of continuous fault‑injection practice across the organization.

System maturity is classified into five levels, guiding the selection of appropriate fault scenarios. Even single‑point services can start with simple experiments such as CPU saturation to validate monitoring and drive multi‑instance deployments.

ChaosBlade, an open‑source chaos‑experiment tool written in Go, supports a rich set of scenarios for applications, containers, and infrastructure. It uses a CLI built on the Cobra framework, follows a four‑stage experiment model, and represents each experiment as a UID‑identified object, enabling easy extension via YAML‑described scenarios.

To control the blast radius, two methods are used: limiting experiment granularity from data‑center level down to individual users, and isolating a subset of production machines with traffic replay to safely execute fault injections.

The AHAS Chaos platform orchestrates experiments in four stages—preparation, execution, verification, and recovery—allowing custom mini‑programs for monitoring integration, notifications, and other extensions. Its architecture exposes OpenAPI and a workflow engine to standardize chaos‑engineering processes.

High‑availability principles for distributed services are enumerated, including load balancing, traffic steering away from unhealthy nodes, rate limiting, timeout retries, dependency degradation, circuit breaking, observability, gray releases, rollback capability, and elastic scaling.

A demo topology consisting of consumer, provider, and base services (with multiple instances) and a MySQL database is used to illustrate the practice. The architecture is automatically discovered by AHAS and visualized in the platform.

Case Study 1 validates monitoring and alerting by injecting a 600 ms delay into MySQL queries for 50 % of requests using ChaosBlade. The ARMS monitoring system promptly triggers a DingTalk alert, confirming the effectiveness of the defined metric.

Case Study 2 tests instance isolation by adding latency to one provider instance. The expected behavior—automatic isolation and quick QPS recovery—did not occur, revealing a gap that was manually fixed by taking the faulty instance offline, after which normal traffic resumed.

The presentation concludes that chaos engineering is a proactive stability technique embodying antifragility, requires steadfast commitment to its principles, must be driven by clear objectives, and depends on appropriate tools and risk‑controlled execution to become a routine practice.

Relevant Alibaba Cloud products include AHAS (architecture awareness, fault‑injection, rate limiting), ARMS (monitoring and tracing), and PTS (performance testing). The ChaosBlade project can be accessed at https://github.com/chaosblade-io/chaosblade.