How to Simulate 100 Billion WeChat Red‑Packet Requests on a Single Server

Background

Original Spring Festival red‑packet system handled 14 million QPS on 638 servers (~23 k QPS per server). The prototype targets a single machine supporting 1 million concurrent connections and 30 k–60 k QPS.

Target Metrics

≥1 million concurrent connections.

≥30 k QPS (goal 60 k QPS).

Shake‑red‑packet request rate ≈83 /s per server (derived from 5 × 10⁴ total per‑second distribution ÷ 600 servers).

Send‑red‑packet throughput 200 packets /s.

Hardware & Software Stack

Software: Go 1.8r3, shell scripts, Python for monitoring. OS: Ubuntu 12.04 (server) and Debian 5.0 (clients). Hardware: Dell R2950 8‑core, 16 GB RAM (non‑exclusive). 17 ESXi 5.0 VMs each with 4 CPU/5 GB RAM act as clients, each establishing 60 k connections to reach 1 million total.

Implementation Overview

Connections are partitioned into multiple SET objects, each managing a few thousand connections. This reduces goroutine count to roughly one per connection plus a small set of worker goroutines. Client request timing is synchronized via NTP timestamps; users are divided into groups so each group sends one request per second.

Server processes three message types:

Shake‑red‑packet request – checks a per‑SET red‑packet queue; if empty, returns “no packet”.

Other client messages (e.g., chat).

Server responses.

Red‑packet generation runs at a low rate (200 /s) to keep the queue populated.

Monitoring

Custom Python scripts combined with ethtool monitor packet rates. Counters are embedded in client and server code and reported to a lightweight monitoring service.

Experimental Procedure

Start server and monitoring, launch 17 client VMs, verify 1 million connections using ss.

Increase client QPS to 30 k via HTTP control; observe stable QPS and red‑packet generation (200 /s).

Raise client QPS to 60 k and repeat the test.

Results: stable 30 k QPS; at 60 k QPS fluctuations appear due to goroutine scheduling, network latency, and occasional packet loss.

Data Analysis

Client‑side and server‑side QPS graphs plotted with gnuplot confirm the prototype meets the 30 k–60 k QPS target, though further code optimisation could improve stability.

Conclusions

The single‑server prototype validates that 1 million users and 30 k–60 k QPS for shake‑ and send‑red‑packet workloads are feasible. Scaling to the full 100 billion request scenario would require roughly 600 such servers, completing the workload in about 7 minutes.

References

GitHub: https://github.com/xiaojiaqi/C1000kPracticeGuide GitHub:

https://github.com/xiaojiaqi/10billionhongbaos