Design and Practice of a High‑Throughput Spring Festival Red‑Packet System Supporting One Million Connections

Background Inspired by a 2015 InfoQ presentation on building a "guaranteed" Spring Festival red‑packet system, the author explores how to reproduce a similar high‑load scenario locally. The goal is to simulate 10 billion red‑packet requests over the four‑hour TV show, targeting 1 million concurrent connections and a peak QPS of 60 000 per server.

Target Calculations Based on 638 servers serving 1.43 billion users, the per‑server user limit is ~2.28 million. Assuming 600 active servers, each should handle ~90 0000 users. The required per‑server QPS is derived from the total 14 million QPS demand, resulting in ~23 000 QPS, with a design target of 30 000–60 000 QPS. Red‑packet issuance rate is set to 83 per second per server (scaled from 5 × 10⁴ total).

Hardware & Software Stack Server: Dell R2950, 8‑core, 16 GB RAM, Ubuntu 12.04. Client: Debian 5.0 virtual machines (17 × ESXi 5.0, each 4 cores, 5 GB RAM) establishing 60 000 connections each. Software: Go 1.8r3, shell scripts, Python for monitoring, and various open‑source tools (ethtool, gprof, etc.).

Implementation Details 1. Million‑connection support : Leveraged Go goroutines and a SET‑based architecture where connections are partitioned into independent groups, reducing goroutine count and lock contention. 2. Client QPS coordination : Clients synchronize via NTP and use a simple modulo algorithm to evenly distribute request timing across 1 million users. 3. Server QPS handling : Embedded counters record per‑second request counts; network traffic is monitored with a custom Python‑ethtool tool. 4. Red‑packet business logic : Servers generate red‑packets at a fixed rate; clients request packets, receiving success or failure responses. Lock contention is mitigated by sharding users into buckets; Disruptor could be used for further optimization. 5. Monitoring : A lightweight monitoring module (derived from the author’s fakewechat project) aggregates client and server metrics and visualizes them using simple Python scripts and gnuplot.

Practice Phases Phase 1: Start server and monitoring, launch 17 client VMs to create 1 million connections, verify connection counts with ss . Phase 2: Increase client QPS to 30 000, observe stable QPS and red‑packet issuance at 200 per second. Phase 3: Raise client QPS to 60 000, note increased variance in QPS and red‑packet acquisition due to network jitter and Go scheduler effects.

Results & Analysis Graphs show three QPS regions (baseline, 30 k, 60 k). At 30 k QPS the system remains stable; at 60 k QPS occasional drops occur, attributed to goroutine scheduling, network latency, and packet loss. Overall, the prototype meets the design goal of supporting 1 million users and ≥30 k QPS, with room for further optimization.

Conclusion The simulated system successfully demonstrates that a single server can handle up to 6 × 10⁴ QPS and one million connections, validating the feasibility of processing 10 billion red‑packet requests within minutes when scaled across 600 servers. The architecture is extensible and highlights key considerations for high‑throughput backend services.