Chaos Engineering: Definition, Principles, and Implementation Steps

By Sun Jun on Testing

With the end of Moore's Law, single‑machine computing performance has reached its limit, while software systems continue to grow in scale and complexity. Consequently, systems are increasingly moving toward distributed architectures. In recent years, the emergence of cloud services and containers has made many distributed systems easier to adopt micro‑service patterns.

Regardless of the myriad distributed technologies, the demand for system reliability remains consistent: distributed systems must be highly available and capable of self‑recovery or graceful degradation even when single nodes or clusters fail.

Despite efforts in sound architecture, high‑quality code, and thorough testing, many distributed systems still fall short of high availability and elasticity. To uncover hidden weaknesses, large software companies such as Google, Netflix, and JD.com have introduced chaos engineering. Typical system weaknesses include external service failures causing cascading failures, inappropriate degradation strategies, and improper timeout mechanisms that lead to infinite retries.

Definition of Chaos Engineering

Chaos engineering discovers system weaknesses by observing behavior changes during controlled fault‑injection tests, then improves the system to enhance reliability and confidence in its ability to withstand uncontrolled conditions. It is not a new concept; traditional disaster‑recovery testing is a form of chaos engineering.

General Implementation Steps

Identify measurable indicators of normal operation as a baseline "steady state".

Assume both experiment and control groups can maintain this steady state.

Inject events into the experiment group (e.g., server crashes, disk failures, network disconnections).

Compare the steady states of experiment and control groups, disproving the assumption if they diverge.

If the steady states remain consistent after fault injection, the system is considered resilient. If they differ, a weakness has been found and can be addressed.

Ideal Principles of Chaos Engineering

1) Form a hypothesis based on the system’s characteristics in a steady state

Using an e‑commerce order flow as an example, the hypothesis focuses on high‑level external metrics such as order volume, transaction amount, throughput, latency, and error rate, rather than individual service internals. However, micro‑level metrics (CPU, I/O, etc.) should also be monitored to catch issues like cache failures that may not affect macro metrics.

2) Choose events that could realistically occur in the real world

Any condition that may affect the steady state qualifies as an event, including hardware failures (server crash, network outage) and software failures (external service unavailability), as well as non‑fault events like traffic spikes.

3) Run experiments in production

Only production metrics are truly predictable (e.g., daily registrations, daily orders). Since test environments cannot fully replicate production, running chaos experiments in production provides a realistic assessment of reliability.

4) Integrate with continuous integration

Internet software is updated daily; therefore, applying chaos engineering continuously, similar to CI pipelines, is practical.

5) Minimize impact scope

Because chaos experiments can cause service outages or financial loss, the impact must be limited and quickly recoverable. Techniques such as A/B testing can help contain the effect.

The above describes the ideal scenario. In practice, implementation is staged according to software maturity.

Stage 1: Basic Distributed System Resilience

Example from JD.com: before the Double‑11 shopping festival, they conduct fault‑drills, dividing teams into fault creators and responders to evaluate detection, response, handling, and recovery capabilities. This intensive pre‑event chaos practice improves tolerance to large‑scale failures.

Example from Youzan: initially, chaos engineering is performed in test environments to control risk. By selecting critical APIs that affect key business metrics (registration, order volume) and running scenario‑based integration tests after fault injection, they assess reliability even without production data.

Chaos engineering can also be viewed as a generic, automated approach to injecting unpredictable anomalies. Manually injecting specific faults and providing corresponding recovery mechanisms enables broader anomaly testing.

Stage 2: Mature Distributed System Resilience

Netflix has largely adopted the ideal steps and principles, running continuous, automated chaos experiments on weekdays, achieving high reliability and elastic scaling.

Youzan’s Chaos Engineering Implementation

Because chaos engineering intentionally introduces faults, we nickname the tool "Weizhen Tian" (a villain from Transformers). At the current early stage, fault injection is manually controlled, and the following fault types have been implemented:

CPU high load

Disk high load: frequent read/write

Disk space exhaustion

Graceful application shutdown via stop script

Forceful kill of processes, potentially causing data inconsistency

Network degradation: corrupt packet data

Network latency: delay packets within a range

Network packet loss: simulate partial TCP loss

Network blackhole: drop packets from a specific IP

External service unreachable: redirect domain to localhost or drop outbound packets

References PRINCIPLES OF CHAOS ENGINEERING (http://principlesofchaos.org/)

Other Articles by the Author Two Testing Methods for Asynchronous Systems

Open‑Source Project – BugCatcher A management tool for product, development, and testing collaboration (https://github.com/youzan/bugCatcher)

PS: Youzan's testing team is hiring. If you are interested, send your resume to [email protected].