How We Scaled a Volunteer Registration System to 20k QPS with Redis, RocketMQ, and MySQL

Background and Requirements

The project was a provincial volunteer‑registration system that suddenly faced severe performance complaints; the client demanded that any future slowdown could jeopardize the company's contracts. The system needed to support massive concurrent traffic while guaranteeing absolutely accurate data.

Analysis of Load Requirements

Candidate‑side login and volunteer‑info query: 40 000 QPS

Candidate‑side save‑volunteer interface: 20 000 TPS

Exam‑info query: 40 000 QPS

Teacher‑side access: 4 000 QPS

Import operations: no hard limit, but must finish within 20 minutes and recover within 20 minutes

Data must be 100 % accurate, with no loss or inconsistency

Initial Experiments with MySQL

Single‑node MySQL was benchmarked by inserting and querying a user table. The best‑case throughput was about 5 k TPS and 12 k QPS, but adding indexes reduced TPS by 1‑1.5 k. Replication and read/write splitting further degraded performance, so a pure MySQL solution was ruled out.

Introducing Redis Cache

Redis was added as a front‑line cache. Single‑node tests showed GET reaching 100 k QPS and SET around 80 k TPS, meeting the raw throughput requirement. However, Redis’s in‑memory nature raised concerns about data loss, prompting a high‑availability design.

Combined Architecture (Redis + RocketMQ + MySQL)

The final design stored all read/write data in Redis, persisted asynchronously to MySQL, and used RocketMQ to guarantee reliable message delivery. The workflow for the critical “save‑volunteer” API became:

Start a Redis transaction and update Redis data

Persist the message synchronously to RocketMQ (sync‑disk mode)

Commit the Redis transaction

Asynchronously write to MySQL

Fault‑Recovery Strategy

MySQL crash: automatic recovery via its own redo logs.

RocketMQ crash: switched from async to sync disk to avoid message loss.

Redis crash: enabled Redis transactions and added a timestamp field; a scheduled task compares Redis timestamps with MySQL and repairs inconsistencies.

Load‑Testing Rounds and Findings

Initial load tests after implementation showed correct data consistency but only ~4 k TPS. Adding more nodes raised TPS modestly; the bottleneck shifted to RocketMQ’s synchronous disk writes. Subsequent analysis identified Redis JSON payload bandwidth as another limiting factor, leading to gzip compression of the payload.

Further Optimizations (compression, partitioning, connection limits)

Compressing the JSON reduced network usage, and oversized strings were split across multiple keys. Horizontal scaling of RocketMQ and Redis hash‑slot partitioning (using the last digits of the ID card number) balanced load. The built‑in Tomcat connector limited concurrent connections to ~1 k; increasing the number of Java service nodes and moving to Nginx VIP load‑balancing raised the effective concurrency ceiling.

Final Deployment and Outcome

After multiple iterations, the system achieved up to 40 k QPS for query interfaces and 20 k TPS for the save‑volunteer interface, with response times under 100 ms. The deployment involved 8 candidate‑side Java nodes, 4 management nodes, 4 RocketMQ brokers, 4 Redis shards, a MySQL master‑slave pair, and an Elasticsearch instance. Post‑deployment monitoring confirmed stability, and the project successfully rescued the company’s contract while providing valuable performance‑engineering experience.