Simulating 10 Billion Red‑Packet Requests on One Server: Achieving 60k QPS with Go

Background

The goal is to reproduce a high‑concurrency scenario similar to WeChat’s red‑packet (shake and send) service, where billions of requests occur during peak events. The author explores whether a single server can support 1 million connections and sustain high QPS.

Capacity Planning

Estimated user base: 2.28 million users per server (based on 14.3 billion total users across 638 servers).

Active users during peak: around 540 million monthly active, with far fewer concurrent online users.

Target per‑server load: ~90 000 users, 2.3 × 10⁴ QPS (scaled up to 3 × 10⁴ and 6 × 10⁴ QPS for testing).

Red‑packet issuance rate: 5 × 10⁴ per second system‑wide, which translates to ~83 per second per server.

System Design

Software stack : Go 1.8r3, Shell, Python; OS: Ubuntu 12.04 (server), Debian 5.0 (client).

Hardware : 8‑core Dell R2950 server with 16 GB RAM (non‑dedicated), 17 virtual client machines (4 CPU, 5 GB RAM each) establishing 60 000 connections per VM to reach 1 million total connections.

Architecture : The server divides connections into multiple independent SET groups, each managing a few thousand connections. A single goroutine per connection reads messages and forwards them to the SET’s queue; each SET has a dedicated worker goroutine handling three message types (shake request, other client messages, server responses). Red‑packet generation is performed by a lightweight service that inserts packets into each SET’s queue in a round‑robin fashion.

Implementation Highlights

Goroutine count reduction: each connection uses one goroutine; SET workers handle message routing, cutting total goroutine count from millions to a few hundred thousand.

Load distribution: SETs can be assigned to servers of varying CPU cores, enabling fine‑grained scaling.

Time‑based request coordination: clients synchronize via NTP and calculate their request slot using if time() % 20 == userID % 20 { send request } to evenly spread load.

Testing Phases

Phase 1 : Start server and monitor, launch 17 client VMs, verify 1 million connections using ss command.

Phase 2 : Increase client QPS to 30 000 via HTTP API, observe stable QPS and red‑packet issuance at 200 packets/s.

Phase 3 : Push client QPS to 60 000, repeat red‑packet generation, and record performance degradation points.

Results

QPS remained stable around 30 k QPS; at 60 k QPS the server showed occasional spikes and drops due to goroutine scheduling, network latency, and packet loss. Monitoring graphs (included) illustrate client and server QPS trends, red‑packet generation rates, and Go pprof data showing occasional GC pauses >10 ms on the older hardware.

Conclusion

The prototype meets the design targets: a single machine can handle 1 million concurrent users and sustain up to 60 k QPS for the shake‑and‑send red‑packet workflow, demonstrating that the system is horizontally scalable and that the core concepts can be applied to larger deployments.

Reference implementations and documentation are available at:

C1000kPracticeGuide

10billionhongbaos