Chaos Engineering in WeChat Pay: Design, Implementation, and Results

WeChat Pay, a national critical information infrastructure serving millions of merchants and hundreds of millions of users, requires availability higher than five nines. To improve resilience, the team introduced chaos engineering, focusing on controlling the minimal blast radius and building an automated fault‑injection system.

The practice follows the five Netflix chaos‑engineering principles: assume a stable state, use diverse real‑world events, run experiments in production, automate experiments, and keep the blast radius minimal. By partitioning the system into multiple independent zones and routing traffic accordingly, experiments can be conducted in isolated chaos partitions without affecting live traffic.

Key challenges include balancing safety (no impact on production) with effectiveness (faults close to real failures). The team evaluated several deployment options, concluding that a multi‑partition architecture provides the best trade‑off, allowing realistic fault injection while maintaining isolation.

Experiment design covers four dimensions: business assets, system resources, fault types, and fault severity. Automation is achieved through YAML‑based experiment orchestration, support for sequential and parallel execution, and scheduled runs. Automatic analysis matches experiment windows with alerts from business, module, and infrastructure monitoring to detect deviations.

Results to date (2021‑2022) include over 60 experiment plans, more than 500 experiment executions, zero production incidents, and the discovery of numerous high‑risk issues in core components, RPC frameworks, queues, storage, and payment flows. The chaos‑engineering platform now supports 30+ fault injectors, a drag‑and‑drop UI for experiment composition, and integration with the SOA governance system for risk tracking.

Future work aims to expand the fault‑atom library, improve automated steady‑state detection with AI, and support a broader range of blast‑radius configurations for non‑core services.