Design and Implementation of QQ Game Spring Festival Red Packet System: Resilience, Overload Protection, and Monitoring

This article presents a detailed case study of the QQ Game Spring Festival Red Packet project, focusing on how the backend services were designed, developed, and operated to ensure high availability during peak traffic.

System Resilience : The system adopts multi‑data‑center deployment with global load balancing (GSLB), Tencent Gateway (TGW), and QZHTTP servers. Disaster‑recovery is achieved through multi‑machine and multi‑data‑center strategies, leveraging L5 for service discovery and automatic fault removal within 1–2 minutes.

Overload Protection : Four key measures are applied—request filtering at the source, early rejection at the access layer, mutual distrust between layers, and user‑friendly error handling. Specific techniques include traffic pre‑loading via CDN, rate limiting (e.g., one request per 5 seconds per user), degradation switches, and discarding messages that exceed queue latency thresholds.

Flexible Availability : The design distinguishes critical user paths (gift list, server selection, gift receipt) from non‑critical ones. Critical paths are protected with redundant services and fallback mechanisms (e.g., using default gift lists when recommendation services fail). Non‑critical paths are gracefully degraded or hidden when failures occur.

Three‑Dimensional Monitoring : Monitoring covers user‑level (synthetic transactions via ATT), business‑level (module‑to‑module calls and internal status using custom alert systems), and machine‑level (CPU, memory, disk, network via TNM2). Alerts are generated for abnormal queue sizes, service latency, and resource exhaustion.

Exercise Validation : Three types of rehearsals were conducted—gray‑scale testing to validate traffic models, pressure testing to verify capacity (up to 80 k requests/s), and fault‑injection testing to confirm that disaster‑recovery and flexible‑availability mechanisms work as expected.

Conclusion : The project demonstrates a comprehensive approach to backend system design that integrates functional development, performance optimization, fault tolerance, and observability, ensuring a reliable user experience even under extreme load conditions.