Redesign of a Large‑Scale Content Ingestion System: From Microservices to a Plugin‑Based Monolith

The article describes the complete refactoring of QQ Browser’s content ingestion system, which originally consisted of more than 90 tiny micro‑services handling thousands of content types. The legacy architecture suffered from low development efficiency, poor CPU utilization (max 40%), excessive RPC calls, duplicated if‑else logic, and fragile fault‑tolerance, leading to long lead‑times for new data sources and severe performance bottlenecks (e.g., processing 6 × 10⁸ documents took 12 days).

To address these issues, the team adopted a “zero‑basis” redesign and migrated to a single‑process monolithic service with a plugin framework. The new design focuses on five key aspects:

Replace fragmented micro‑services with a monolithic service that keeps data in memory, eliminating most RPC overhead.

Introduce a plugin system to encapsulate each functional step (access, processing, distribution), allowing flexible composition without hard‑coded if‑else branches.

Support both incremental updates and batch “刷库” (full‑rebuild) flows by configuring four distinct processing pipelines.

Implement robust fault‑tolerance using Kafka as a message buffer; HTTP/TRPC pushes are first written to Kafka, ensuring no data loss during node failures.

Separate consumption and computation threads via a lock‑free queue, allowing each Kafka partition to be consumed by a dedicated thread while multiple worker threads process the data, dramatically improving CPU utilization and enabling horizontal scaling.

Key performance gains after the migration include:

Average single‑core processing QPS increased from 13 to 172 (≈13× improvement).

Batch processing QPS rose from 13 to 230 (≈17× improvement) and cluster‑wide QPS grew from ~1 000 to 10 000 (≈10×).

Average processing latency dropped from 2.7 s to 0.8 s (‑71%).

p99 latency reduced from 17 s to 1.9 s (‑88%); p999 latency from 19 s to 3.7 s (‑80%).

CPU utilization rose from ≤40 % to near 100 %.

Code‑level refactoring also yielded substantial benefits:

Adopted table‑driven programming to replace long if‑else chains.

Converted switch‑case logic into factories, improving extensibility.

Applied the Single‑Responsibility Principle, moving from monolithic functions to polymorphic classes.

Replaced repeated JSON serialization with rapidjson::Document objects kept in context, eliminating redundant parsing.

Switched from RapidJSON to Sonic‑JSON, gaining ~40 % faster parsing and ~15 % higher throughput.

Integrated jemalloc and tuned malloc_trim to mitigate memory‑pool growth caused by RapidJSON’s MemoryPoolAllocator.

Development processes were also overhauled: CI/CD pipelines were optimized by reducing lock granularity, using GitHub mirrors for faster dependency fetching, and enforcing coding standards, code reviews, and documentation. As a result, the P80 lead‑time for business requirements dropped from 5.72 days to ≤1 day (‑82%), code‑CC issues fell from 568 to 0, unit‑test coverage rose to 77 %, average cyclomatic complexity fell from 24 to 2.31 (‑90 %), total code lines shrank from 113 k to 28 k (‑75 %), and the number of critical service links decreased from 15 to 3 (‑80 %).

Overall, the redesign demonstrates how consolidating micro‑services into a well‑structured monolith with plugin extensibility, modern C++ practices, and robust fault‑tolerance can dramatically improve both system performance and developer productivity.