How Service Degradation and Fault‑Tolerance Keep Large‑Scale Systems Resilient

When a gift‑package request passes through more than ten backend CGI stages, core steps such as reading the package configuration must never be skipped, while non‑core steps like data reporting can be bypassed by assigning them a short timeout. For example, a statistics reporting service averaging 3 ms is given a 20 ms timeout; if it exceeds this, the request proceeds without waiting.

Service Decoupling and Physical Isolation

Designing services to be as small and independently deployed as possible reduces coupling, so a failure in one module affects fewer others, improving fault tolerance. Although early‑stage traffic may be low and resources limited, as traffic grew from millions to hundreds of millions of daily requests (a 100× increase), the pain of tightly coupled services became evident.

Consequently, core services and storage were gradually split into many smaller, independently deployed units. For instance, original 3‑5 storage services expanded to over 20 separate deployments, and one core storage was divided into three parts in late 2016.

The benefits include:

Reduced pressure on the primary storage through load distribution.

Higher stability because a single component failure no longer brings down the entire module.

Physical isolation of storage ensures that hardware failures in one server do not affect others.

Light‑Heavy Separation

Critical business functions are separated by workload intensity. In the 2016 Spring Festival red‑packet activity, the information‑query cluster (light) and the red‑packet issuance cluster (heavy) were deployed independently.

Advantages of this deployment:

If the query cluster fails, the issuance cluster continues to operate, preserving core user functionality.

Both clusters have similar machines and services, allowing mutual support and failover during emergencies.

Machines are distributed across multiple data centers (e.g., A, B, C). If one data center loses network connectivity, the remaining centers continue serving traffic.

Business‑Level Fault Tolerance

Beyond architectural fault tolerance, business‑level errors—especially human mistakes like misconfiguring daily gift limits—can cause major incidents. Monitoring can detect such anomalies quickly, but even short‑lived issues can affect thousands of users at large scale.

To prevent these errors at the source, a robust, intelligent configuration‑checking system was built, aggregating dozens of business rules that validate simple limits (e.g., daily gift caps) as well as complex inter‑parameter dependencies. The system enforces checks programmatically, eliminating reliance on informal “oral agreements” and ensuring that activities are fully verified before launch.

Conclusion

Fault tolerance not only strengthens system robustness but also frees engineers from constant emergency alerts, allowing more stable and predictable operations. Achieving full resilience remains an ongoing journey.