Design and Optimization of a High‑Throughput Messaging Platform
To handle exploding daily traffic, the team rebuilt the messaging platform with a unified API, concurrent consumption, MongoDB storage, and a priority scheme that uses separate Kafka topics and adjustable pull ratios, while employing a state‑machine‑driven thread pool, multi‑tier delay mechanisms, and MongoDB/Redis‑based fatigue control, delivering fast, traceable, hierarchical urgent delivery with visual metrics and content safety.
Background: Rapid growth increased daily message volume, leading to a rebuild of the messaging system.
Legacy issues: slow integration and sending, no traffic analysis, no priority handling, and blocking of critical messages during bulk pushes.
Optimizations: unified API with business identity, concurrent consumption, priority support, and switching storage to MongoDB for higher write throughput.
Priority: Kafka lacks native priority, so separate topics per level are used and consumption ratios adjusted. A state machine tweaks pull parameters, and a local thread pool with a priority task queue handles urgent messages.
Delay: three strategies (<30 min, <15 s, >30 min) use Java DelayTask, a second‑level time wheel, and a hybrid method that stores long‑term tasks and moves them to the wheel before execution.
Fatigue control: MongoDB stores aggregation data; Redis HyperLogLog handles per‑business daily limits.
Takeaways: simple integration, fast response, hierarchical urgent delivery, traceability, visual metrics, and content safety.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
DeWu Technology
A platform for sharing and discussing tech knowledge, guiding you toward the cloud of technology.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.