An Introduction to Chaos Engineering: Principles, Practices, and Tools

Chaos History

Software and system development is an exercise in innovation and solving unknown problems, and it is prone to errors because it is built by people with diverse viewpoints and skills, often in large teams. As technology becomes more distributed and complex—especially with the rise of micro‑services—few individuals possess complete end‑to‑end knowledge of an entire system.

Similar to the military term "fog of war," understanding the overall impact of changes in modern development can be difficult, creating a "development fog." Coupled with user expectations for continuous availability, testing system robustness and resilience to unknowns becomes a critical challenge.

Chaos engineering addresses this by injecting faults throughout the application and infrastructure stack, allowing engineers to observe behavior, verify resilience, and adjust systems so that failures never surface to users. The rise of Site Reliability Engineering (SRE) practices further emphasizes the need to quantify unlikely impacts.

What Is Chaos Engineering?

Chaos engineering is the science of deliberately injecting failures into a system to measure its resilience. Like any scientific method, it focuses on hypothesis‑driven experiments and compares results against a steady‑state baseline. A typical example in a distributed system is randomly shutting down a service to see how the overall application responds and where user journeys are impacted.

If you take a cross‑sectional view of what an application needs to run—compute, storage, network, and application infrastructure—injecting faults or turbulent conditions into any part of that stack constitutes a valid chaos experiment. Known failure modes such as network saturation or sudden storage instability can be safely tested, and virtually any team that supports the stack can be a stakeholder.

Who Uses Chaos Engineering?

Because chaos engineering touches many technologies and decisions, experiments can involve multiple stakeholders. The larger the blast radius (the scope of impact), the more stakeholders are involved.

Stakeholders vary based on the domain of the application stack—compute, network, storage, or application infrastructure—and the location of the target infrastructure.

If the blast radius is small enough to run inside a container, the application development team can conduct the test without involving others. When the impact spans larger workloads or infrastructure (e.g., testing Kubernetes), platform engineering teams typically participate. Providing coverage for the unknown is the core reason for running chaos tests and finding weaknesses.

Why Run Chaos Tests?

The "development fog" is especially real for large, distributed, or micro‑service‑based systems. From an application perspective, each micro‑service can be tested in isolation to confirm it works as designed, and normal monitoring may deem a single service healthy.

However, a single request often traverses multiple services to produce an aggregated response. Each remote call adds additional infrastructure and crossing of application boundaries, any of which can fail.

If a trivial or critical service fails to respond within its Service Level Agreement (SLA), the entire user journey is affected—this is precisely the problem chaos engineering aims to solve, using experiment results to build more resilient systems.

Chaos Engineering Principles

The "Principles of Chaos Engineering" article outlines four scientific‑method‑like practices. Unlike the classic scientific method, chaos engineering assumes the system is stable and then seeks variance. The harder it is to break the steady state, the higher the confidence in the system's robustness.

Start with a Baseline (Steady State)

Understanding what normal looks like is essential for detecting deviation. Metrics such as response time or the ability to complete a user journey within a time window serve as good indicators of normalcy. The steady state acts as the control group in experiments.

Assume the Steady State Will Persist

Contrary to scientific hypothesis, assuming a condition will always hold leaves little room for discovery. Chaos engineering targets robust, stable systems to uncover hidden failures. Running chaos experiments on already unstable systems provides little value because their unreliability is already known.

Introduce Variables/Experiments

As with any experiment, chaos engineering introduces variables to see how the system reacts. These variables represent real‑world fault scenarios affecting one or more of the four pillars: compute, network, storage, and application infrastructure. Faults can be hardware failures, network partitions, etc.

Attempt to Refute the Hypothesis

If the hypothesis concerns the steady state, any variance or interruption between control and experiment groups refutes the stability assumption. This highlights areas for repair or design changes to make the system more robust.

Chaos Engineering Best Practices

Three pillars guide effective chaos engineering: provide sufficient coverage, run experiments continuously (especially in CI/CD pipelines), and minimize the blast radius.

Provide Coverage for Estimated Failure Frequency/Impact

Achieving 100% test coverage is impossible; instead, focus on the most impactful scenarios—e.g., storage unavailability or network saturation—that could cause severe disruption.

Run Experiments Continuously in Your Pipeline

Software, systems, and infrastructure change rapidly. CI/CD pipelines are ideal for automatically executing chaos experiments whenever code changes occur, building confidence in the system before release.

Run Experiments in Production

Testing in production exposes the system to real traffic and load, providing the most accurate insight into resilience.

Minimize Blast Radius

Responsible chaos engineering limits the blast radius to small, focused experiments—such as injecting latency between two services—so that failures are contained and insights are actionable.

Chaos Engineering vs. Load Testing

Load testing ensures a system can handle expected traffic by scaling resources, but it does not reveal how the system behaves when a component is missing or fails catastrophically. Chaos engineering fills this gap by exposing hidden single‑point failures and cascade effects.

Chaos Engineering Tools

There are many tools and platforms for chaos engineering. Notable examples include:

Chaos Monkey

One of the earliest chaos engineering tools, created by Netflix, originally disabled random production instances and later became part of the Simian Army.

The Simian Army

A suite of tools (e.g., Janitor Monkey, Latency Monkey, Security Monkey, Doctor Monkey) that introduced different fault types. The project has since been retired.

Gremlin Platform

One of the first SaaS chaos engineering platforms, offering coordinated fault injection experiments and extensive learning resources.

AWS Fault Injection Simulator (FIS)

A managed service for injecting faults into AWS environments, suitable for teams fully invested in the AWS ecosystem.

Regardless of the tool chosen, CI/CD pipelines remain an excellent place to orchestrate chaos experiments.

Testing Your CI/CD Pipeline

Modern confidence‑building practices encourage running chaos experiments directly in CI/CD pipelines. Recent demos from Harness and Gremlin show how to integrate experiments into deployment workflows.

One possible pattern is to let experiment results influence deployment decisions, or to run experiments in lower environments before promoting to production.

Harness Helps Here

Harness is a software delivery platform that coordinates confidence‑building steps. For teams new to chaos engineering—especially those managing dozens of applications they did not write—Harness can isolate or spin up new releases for experimentation without impacting production.

If your applications are not deployed through a robust pipeline, creating an isolated deployment can be painful; integrating chaos experiments into the workflow simplifies this process.

Note: The original source contains promotional links and community references that have been omitted from this academic summary.