Designing High‑Concurrency Distributed Push Systems: Lessons from Meizu’s Backend Architecture

Concurrency vs. Distribution

Concurrency describes the number of requests a single machine can process, while distribution involves multiple machines cooperating to increase capacity and eliminate single‑point failures.

Core Practices for High‑Concurrency Distributed Systems

Decompose a monolithic system into many small, independent services to reduce complexity.

Adopt a mature communication framework (e.g., RPC) to improve development efficiency.

Prefer multi‑process or lock‑free designs over multi‑threading to avoid lock contention and deadlocks.

Implement overload protection, comprehensive monitoring, and gray‑release (canary) mechanisms.

Why C++ Was Chosen

High performance for millions of concurrent connections.

Broad talent pool makes recruitment easier.

Fine‑grained memory control, essential when handling tens of millions of users where uncontrolled memory growth can be catastrophic.

Long‑Connection Heartbeat Management

Mobile clients maintain a persistent TCP connection and send periodic heartbeat messages. Each heartbeat carries the interval for the next heartbeat. The server reads this interval and dynamically sets the connection timeout, balancing power consumption on the device with network traffic on the server.

Evolution of the Real‑Time Push System

Access‑Layer Refactor – The original single server handled up to 300,000 long connections and also executed business logic, creating a performance bottleneck and tangled code. The layer was split: the front‑end now only accepts connections and forwards requests, while all business logic runs in backend microservices.

Bandwidth Optimization – A custom binary protocol replaced the initial text‑based protocol, cutting bandwidth usage by 50‑70% during mass push operations.

RPC Framework Development

Version 1 : Multi‑threaded synchronous calls. Resulted in heavy lock contention and occasional deadlocks.

Version 2 : Multi‑process asynchronous calls. Achieved ~10× performance gain but introduced complex callback chains and memory‑leak risks (e.g., allocated buffers passed to callbacks without proper release).

Version 3 : Coroutine‑style design inspired by Go. Provides synchronous‑like code flow with the performance of asynchronous I/O, eliminating callback hell and reducing memory‑leak surface.

Storage Enhancements

Introduced Redis for caching and a Redis cluster to eliminate single‑point failures and increase capacity.

Adopted MongoDB for message persistence. Sharding was manually configured with carefully chosen shard keys, balancer scheduled during low‑traffic periods, and indexes tuned to query patterns.

Practical Advice for New Engineers

Maintain strong technical focus and curiosity.

Study high‑quality open‑source projects such as Nginx and Redis to understand production‑grade design patterns.

Participate in technical forums and community events to broaden perspective.

Continuously practice, reflect on code, and iterate on designs.