Technical Strategies for Ensuring System Stability During Large‑Scale Promotional Events

Background

Large‑scale sales promotions (e.g., the 618 event) generate roughly 10% of annual GMV, with peak sales reaching up to 1,463 million CNY per minute in 2022, making system stability a critical requirement.

Challenges

Traffic magnitude : traffic can be dozens of times higher than normal, turning minor issues into major outages.

Data volume : billions of orders create heavy query loads.

Complex scenarios : numerous promotional rules, platforms, and merchants keep the order pipeline under constant high load.

Long delivery chain : multiple services (traffic distribution, promotion calculation, cart, settlement, payment, logistics, customer service) must stay available; a 99.9% availability per service results in only ~99.5% overall.

Low tolerance : users expect flawless experience, leaving little room for errors.

Stability Assurance

The following measures are recommended from three perspectives: application, storage, and operations.

Application perspective

Modular deployment : split the system into independent units to limit failure impact, simplify troubleshooting, and enable independent scaling.

Monitoring & alerting : define granular monitoring layers (middleware, RPC, method, machine, system, business, process, dashboard), set appropriate sensitivity, ensure coverage of key metrics (CPU, latency, traffic, limits, exceptions), and establish rapid alert handling procedures.

Log management : standardize log format, level, output, archiving, and trace‑ID strategy while suppressing noisy logs.

Fast failure : configure dynamic thread‑pool timeouts, leverage middleware timeout features, and apply service‑level rate limiting.

Rate limiting : base limits on system capacity, use per‑service or global limits, and prioritize critical services during spikes.

Business degradation : design fallback paths that sacrifice non‑core features to preserve core functionality.

Storage perspective

Database : adopt primary‑secondary architecture, read‑write separation, appropriate transaction isolation, sharding, and slow‑query optimization.

Cache : use master‑slave clusters, scale shards before hitting capacity, enable multi‑read, handle hot keys, and avoid large keys that degrade performance.

Elasticsearch : deploy dual clusters for redundancy, monitor and kill slow requests, throttle write traffic, and watch storage watermarks.

Operations perspective

Dedicated war‑room team : ensure rapid response, coordinate impact assessment, and enforce process compliance.

Stress testing : conduct realistic traffic simulations to calibrate monitoring thresholds, rate‑limit values, and scaling plans.

Technical freeze : limit code releases during the promotion, as most incidents stem from deployments.

Daily checks & on‑call duty : perform scheduled health checks to detect issues early.

Emergency response plan : define fallback procedures for rapid issue containment and recovery.

Conclusion

The article provides a comprehensive technical roadmap for preparing and safeguarding systems during high‑traffic promotional events, emphasizing that successful stability relies on thorough pre‑planning, cross‑team collaboration, and continuous operational discipline.