How to Build a 100‑Billion Red‑Envelope System that Handles 60 k QPS

Background

Goal: a single machine supports 1 million connections, simulates shaking and sending red envelopes, with a peak QPS of 60 k and stable business load.

Key Concepts

Definitions of QPS (queries per second), PPS (packets per second), shaking red envelope (client request to obtain a red envelope if available), and sending red envelope (generating a red envelope and distributing it to designated users).

Determine Targets

Based on 638 servers handling 1.43 billion users, the per‑machine user limit is approximately 2.28 million; realistic concurrent users per server are about 90 k.

Load Calculations

Assuming 100 billion shakes over a 4‑hour event, per‑machine QPS ≈1 157 (≈1 k /s). The target peak QPS for the whole system is 14 million, which translates to about 23 k /s per machine; the design sets goals of 30 k–60 k QPS.

System Requirements

Support ≥1 million connections.

Process ≥23 k QPS (target 30 k–60 k).

Shake red‑envelope rate ≥83 per second (≈2.3 k QPS).

Send red‑envelope rate ≥200 per second.

Hardware & Software

Server: Dell R2950, 8‑core CPU, 16 GB RAM, Ubuntu 12.04. Client: Debian 5.0 on ESXi VMs (17 machines, each 4 core, 5 GB RAM, 60 k connections). Software stack: Go 1.8r3, Shell scripts, Python for monitoring.

Implementation Highlights

Connections are partitioned into multiple independent SETs, each managing a few thousand connections, reducing goroutine count and lock contention. Clients synchronize time via NTP and decide when to send requests using the formula time()%20 == userID%20, which evenly distributes 5 k QPS across 1 million users.

Monitoring

A Python script combined with ethtool monitors packets per second; counters are sent to a lightweight monitoring service (no opentsdb).

Red Envelope Logic

The server generates red envelopes at a fixed rate and places them in per‑SET queues. When a client shakes, the server checks the queue; if an envelope is available it is returned, otherwise a “no envelope” response is sent. Users are bucketed to reduce lock contention, and a high‑performance queue such as Disruptor is suggested for further scaling.

Testing Phases

Phase 1: launch the server and 17 client instances, establish 1 million connections, verify with ss command.

Phase 2: set client QPS to 30 k, observe network and server metrics, run a red‑envelope generator at 200 /s (total 40 k). Monitoring shows QPS reaching the expected level.

Phase 3: increase client QPS to 60 k, repeat generation and distribution tests. Graphs show higher variance but still meet the 200 /s envelope distribution target.

Results

Client QPS graphs display a stable 30 k region with minor fluctuations; at 60 k QPS the curve shows more jitter due to goroutine scheduling, network latency, and occasional packet loss. Server QPS mirrors client behavior, confirming the design’s scalability.

Conclusion

The prototype demonstrates a backend capable of handling 1 million users and up to 60 k QPS per machine, effectively simulating WeChat’s red‑envelope feature, with clear avenues for further optimization such as reducing GC pauses and employing more efficient queues.