How Tencent Game Teams Use Chaos Engineering to Boost Reliability and Reduce Outages

Definition of Chaos Engineering

Chaos engineering deliberately injects faults into a system to expose hidden weaknesses, verify that monitoring and recovery mechanisms work, and reduce the probability of production failures.

Benefits

Pre‑emptive fault prevention by fixing issues before they appear in production.

Accelerated fault detection through injected anomalies.

Improved fault‑response speed by running experiments after work hours.

Enhanced fault localization using observability tools.

Verification of failover, circuit‑breaker and degradation strategies.

Systematic post‑mortem analysis with complete experiment data.

Platform Architecture and Experiment Lifecycle

The chaos platform supports the full experiment lifecycle: design, execution, and reporting. It provides a library of fault atoms, target selection, orchestration workflows, real‑time metric collection, automatic protection, and persistent storage of results.

1. Experiment Design (Pre‑experiment)

Users choose fault atoms (e.g., CPU load, network latency, pod deletion), define targets such as Kubernetes clusters, IP ranges or physical machines, and compose orchestration workflows via a drag‑and‑drop UI.

2. In‑experiment

During execution the platform injects the selected faults while continuously collecting infrastructure metrics (CPU, I/O) and business metrics (QPS, latency, concurrent users). If predefined steady‑state thresholds are breached, a hook automatically aborts the experiment.

3. Post‑experiment

After completion the platform generates a detailed report, aggregates historical data, stores it for later analysis, highlights new risks, suggests remediation actions and assigns owners.

Fault Atoms

Storage layer: I/O high load, latency, errors, file‑handle exhaustion.

Compute layer: CPU high load, full utilization.

Network layer: latency, packet loss, out‑of‑order, duplication, bandwidth saturation, port exhaustion.

Node/Container layer: host shutdown, pod deletion, container kill.

Application layer: process crash, HTTP status‑code errors.

Custom: user‑provided shell/Python scripts or Go binaries for specialized scenarios.

Key Technologies

The platform combines a self‑developed chaos engine with open‑source solutions such as https://github.com/chaos-mesh/chaos-mesh to provide a rich set of fault atoms for Kubernetes environments.

Experiment Orchestration

Experiments are defined through form‑based configurations. For example, a user can specify a CPU load of 80 % for 10 minutes or inject a 1‑second network delay for the same duration; the platform executes the plan automatically.

Observation and Metrics

The platform integrates with existing monitoring systems (e.g., Prometheus) and can ingest custom business metrics. Real‑time dashboards display the impact of injected faults on both infrastructure and user‑facing KPIs.

Risk Control and Automation

Large‑scale production chaos drills are performed roughly every six months, with most experiments run in pre‑release environments that mirror production. Automatic protection stops experiments when steady‑state metrics cross configured thresholds, preventing uncontrolled outages.

Red‑Blue War Games

Teams conduct adversarial exercises where one group attacks another’s services using the chaos platform. The results expose reliability gaps and drive continuous improvement in incident response and system design.

Practical Experiments Conducted

Single‑point failures (machine, pod, container termination).

Alert validation (triggering and handling alerts).

Strong/weak dependency discovery.

Network jitter and packet loss simulations.

Data‑center outage drills.

Third‑party service degradation tests.

Overload protection and rate‑limiting verification.

Automation Integration

Chaos experiments are integrated into the CI/CD pipeline so that each version release automatically triggers a predefined chaos test suite, reducing manual effort and ensuring consistent coverage.

Application‑Level Fault Injection via Gateway

Beyond infrastructure faults, the platform can inject application‑level faults through a service‑mesh gateway. It can modify HTTP status codes, add response delays, alter headers, limit bandwidth, or filter users, enabling fine‑grained fault injection that affects only targeted player groups.

Experiment Reporting and Data Persistence

All experiment metadata, orchestration configurations, and steady‑state metrics are persisted. Reports include identified risks, root‑cause analysis, remediation owners, and historical trends, enabling a closed‑loop improvement process.

Observed Benefits

Automation reduced the time to run a full chaos test from hours to minutes, accelerated fault detection, increased overall system reliability, and lowered operational costs.