How a Structured Refactor Cut Game Service MQ Calls by Up to 80%

Background

The game business launched in 2017 accumulated many MQ consumers for product‑update messages. Over time 19 consumers across different clusters performed duplicate queries and updates, generating new messages and causing the downstream RPC interface to exceed its 600k/min rate‑limit, even though the raw update traffic was only about 3k/min.

Origin of the Issue

An alert triggered when the downstream RPC call rate hit the limit. Investigation revealed that external update operations, though low in volume, caused a cascade of internal MQ consumption, leading to exponential growth in interface calls.

Pre‑Refactor Situation

Each of the 19 consumers contained its own query and update logic; some even produced new messages, further inflating traffic. Deprecated consumers were still running and duplicate logic existed.

a. Logic scattered, poor maintainability
b. Service call volume multiplied
c. Concurrent updates causing overwrites
d. Deprecated or duplicate consumers still running

Problem Analysis

Rapid feature iteration encouraged adding new consumers for speed, but the growing number made the system hard to understand and maintain. Two core goals were defined: reuse data to reduce queries and suppress unnecessary update calls.

Refactor Goals

The refactor aimed to achieve a reasonable structure, eliminate redundant consumption, improve throughput, and establish a low‑coupling architecture.

a. Reasonable structure
b. Optimize duplicate/invalid consumption
c. Increase consumption capacity
d. Logical optimization
e. Build a new system

Solution Design

Architecture Design

The new design leverages the Flyweight and Strategy patterns and consists of three layers.

a. Data pre‑processing
b. Handler‑based consumption by category
c. Consolidated update calls

Data pre‑processing filters and batch‑queries data, removing non‑game messages and reducing repeated queries.

Handler layer extracts common logic into shared handlers and isolates category‑specific logic, improving cohesion and maintainability. A Manage layer under each handler implements the actual consumption.

Update consolidation batches product‑update calls to cut the number of generated messages, directly addressing the call‑rate explosion.

Implementation Plan

Phase 1 – Refactor non‑core business MQ logic, release grayly to verify stability.

Phase 2 – Migrate core business MQ, monitor impact, complete full migration.

Phase 3 – Fine‑tune the new structure, further decouple modules, and achieve the original design goals.

Testing Plan

Black‑box testing to compare data consistency between old and new flows.

White‑box testing for code‑coverage verification.

Pre‑call data comparison by logging parameters of update‑interface calls.

Detailed Design Highlights

Idempotent Gray‑Scale Handling

Spring AOP creates a custom @Idempotent annotation. The aspect checks a cache keyed by msgId; if a message has been processed successfully, the consumer skips re‑processing, preventing duplicate consumption.

Failure Recovery

Failed updates are retried using a RocketMQ‑based retry component. Unsuccessful messages are re‑queued for asynchronous handling, ensuring eventual consistency.

Data Isolation and Monitoring

Each new consumer runs in its own thread‑pool, facilitating isolated monitoring and higher concurrency. Rich monitoring dashboards and enterprise‑WeChat alerts provide real‑time visibility of the new flow.

Results

After launch, downstream core‑interface call volume dropped dramatically, with overall reductions between 50 % and 80 % and an 80 % cut in update‑type calls. Query‑type calls also fell by roughly 50 %.

Clearly define the refactor’s motivation and scope.

Thoroughly understand the existing system before redesign.

Establish solid conventions and a scalable architecture for future growth.