Introduction to Chaos Engineering and Its Practical Exercise Workflow

1. What is Chaos Engineering

Chaos engineering is a systematic approach that deliberately injects faults into a system to observe its behavior under stress, identify hidden weaknesses, and develop optimization strategies, thereby enhancing system stability and preventing unexpected failures.

1.1 Definition

It creates fault scenarios to proactively discover problems before they occur in production.

1.2 Why Conduct Chaos Drills

With the widespread adoption of micro‑services, distributed architectures, and containerization, system complexity and inter‑service dependencies increase dramatically, making any single component’s abnormal change potentially cause cascade effects; chaos drills help uncover fragile links and strengthen them, improving high‑availability and emergency response capabilities.

1.3 Value of Chaos Drills

They validate a system’s ability to withstand disturbances, identify unknown risks early, and ensure the system can resist uncontrolled conditions in production, thereby boosting overall stability.

2. Chaos Drill Practice

2.1 Drill Process Overview

The practice uses JD Cloud RPA automation platform. The red team (attackers) randomly selects a time window and injects faults such as 100% CPU usage, network latency, or JSF interface delay. The blue team (defenders) monitors alerts, diagnoses issues, and performs recovery actions.

Red Team Steps

Create drill plan via the RPA platform’s tool market.

Configure execution environment, select target application and instance IP.

Execute the drill during the scheduled window after approval.

Blue Team Steps

Investigate alerts to locate the faulty instance.

Apply recovery measures, such as restarting services, to restore normal performance.

2.2 Initial Drill Practice

Preparation Phase : Define objectives, select scenarios, applications, and machines, generate a drill plan, and inform relevant personnel. Risk assessment is crucial; early drills may involve simple faults like high CPU or memory, while later stages introduce network latency or process termination.

Execution Phase : Inject faults, monitor logs and metrics. Example: a JSF interface delay of 100 ms (timeout 50 ms) results in 100 % failure rate during the injection period.

Recovery Phase : Detect and locate faults via alerts, restart services, and verify that availability and performance indicators return to normal.

Review Phase : Identify improvement points, such as delayed alarm emails for CPU overload and missing failure‑threshold alerts for JSF timeouts, and update alerting strategies accordingly.

3. Practical Details

3.1 Typical Drill Scenarios – The platform provides ready‑made scenarios that reduce learning cost and increase efficiency.

3.2 Important Assessment Metrics – After a drill, record process steps and metric changes, focusing on the timeliness of fault discovery, localization, and recovery, as well as overall fault tolerance and alert coverage.

3.3 Risk Control – To limit potential damage, control the scope of drills, conduct thorough risk assessments, and implement preventive measures such as multi‑channel alerts (phone, DingTalk) and defined failure thresholds.

Conclusion

By simulating real‑world anomalies through chaos drills, teams can uncover hidden issues early, enhance high‑availability, and strengthen emergency response capabilities.