Chaos Monkey and the Simian Army: Building Resilient Cloud Systems

Cloud computing is fundamentally about redundancy and fault tolerance; because no component can guarantee 100% uptime, architects must design systems that remain available even when individual parts fail.

Netflix engineers therefore aim to be stronger than their weakest link, employing techniques such as graceful degradation and redundant deployment across nodes, racks, data centers, availability zones, and regions, while constantly testing their ability to survive rare failures.

They illustrate this with the analogy of regularly practicing tire changes on a deflated wheel to ensure readiness for real emergencies, a practice that is costly in the physical world but cheap and automatable in the cloud.

This philosophy led to the creation of Chaos Monkey , a tool that randomly disables production instances to verify that services can survive such failures without impacting customers; the name evokes a weapon‑armed monkey wreaking havoc in a data center.

Running Chaos Monkey during weekdays under close monitoring allows engineers to learn system weaknesses and build automated recovery mechanisms, so that a failure at 3 am on a Sunday may go unnoticed.

Inspired by Chaos Monkey’s success, Netflix expanded the concept into a virtual "Simian Army" of specialized monkeys that inject various faults and detect anomalies, helping keep the cloud environment safe, reliable, and highly available.

Latency Monkey : introduces artificial latency in RESTful client‑server communication to simulate service degradation and measure upstream response, enabling testing of fault tolerance without shutting down actual instances.

Conformity Monkey : identifies instances that violate best‑practice configurations (e.g., not belonging to an auto‑scaling group) and terminates them, prompting owners to remediate.

Doctor Monkey : runs health checks on each instance, monitors external health signals such as CPU load, and removes unhealthy instances after giving owners time to investigate.

Janitor Monkey : scans for unused resources in the cloud and cleans them up to prevent waste and chaos.

Security Monkey : extends Conformity Monkey to discover security violations or misconfigurations (e.g., improper AWS security groups) and terminates offending instances while ensuring SSL/DRM certificates remain valid.

10‑18 Monkey : tests internationalization and localization by exercising services with different languages and character sets across geographic regions.

Chaos Gorilla : simulates an entire Amazon availability zone outage to verify that services automatically rebalance to functional zones without user impact.

The growing Simian Army gives Netflix confidence in handling inevitable production failures and minimizing user impact, while inviting talented engineers to further optimize and expand these resilience tools.