Introduction to Chaos Engineering – Part 2: Four Steps for Disrupting Complex Systems

Chaos engineering is a discipline for experimenting on distributed systems to build confidence that they can withstand uncontrolled conditions in production, but it is not a silver bullet that automatically fixes or solves problems.

The difficulty lies not in injecting failures, which tools make easy, but in deciding where and why to inject them, requiring deep system understanding and strong observability.

Four key steps are presented:

Set observable metrics: Ensure you can reliably collect the data you care about (e.g., CPU load, request latency) without the measurement itself affecting the system.

Define steady‑state: Use the observable metrics to establish a baseline of normal behavior that can be compared against during experiments.

Form hypotheses about events: Turn intuition about system behavior into testable hypotheses (e.g., "killing one machine will not increase latency").

Run the experiment and validate the hypothesis: Execute the fault injection, observe results, and confirm or refute the hypothesis, learning regardless of outcome.

Examples illustrate how to design simple experiments, such as shutting down one power supply in a data center and checking if a website remains operational, and emphasize keeping experiments as simple as possible while ensuring they are meaningful.

The article also warns that chaos engineering complements, rather than replaces, existing testing methods, and that successful experiments depend on accurate observability, well‑defined steady‑state, and thoughtful hypothesis formulation.

In summary, by following these four steps, teams can systematically discover hidden weaknesses, improve system reliability, and gain confidence that their services will continue to function under real‑world failures.