Microservice Implementation Experience in iQIYI Bubble Backend System

iQIYI’s Bubble platform, a fan‑community product with more than 60 million daily active users, has evolved from a simple social chat service to a large‑scale entertainment community supporting multiple business lines such as明星 (star), video, game, and comic circles. The rapid growth in DAU and QPS (over 100K/s) prompted the team to rethink the monolithic architecture and adopt microservices.

Pre‑microservice situation : Initially, the Bubble backend, operation backend, and feed system shared a single codebase and storage resources. Any change in one module could interfere with others, and the monolithic project could not keep up with the increasing user load. The team split the code into three separate projects (Bubble backend, operation backend, feed backend) and decoupled storage, which reduced deployment risk but increased resource concentration and operational cost.

Microservice principles :

Loose coupling and high cohesion – each service should be independent; data belonging to an entity (e.g., user, circle) lives only in its own service.

Performance awareness – microservice boundaries must be justified by business needs; RPC is preferred over HTTP for high‑QPS internal calls.

Ownership – teams own their services end‑to‑end, improving development quality and incident response.

Technology choices : The team evaluated several RPC frameworks (Solar, gRPC, Dubbo, Spring Cloud) and finally selected the internally‑maintained RPCHUB framework built on Thrift, because it had proven stability in iQIYI’s mobile backend for over a year and the team possessed deep expertise.

Service decomposition :

By business entity – services were created for circles, users, events, resource slots, etc., each exposing a unified RPC interface.

By business function – modules such as sign‑in and circle‑friend relationships were split into independent services to isolate failures and improve scalability.

Governance and fault tolerance : The architecture avoids treating SOA/ESB as a microservice solution; instead, services maintain their own contracts and communication patterns. Fault‑tolerance mechanisms include asynchronous processing, circuit‑breaker (Hystrix), and eventual consistency (retry, task tables, and selective use of TCC). Critical data (e.g., user points) is cached only when necessary, and most data is accessed directly via service calls.

Tooling evolution : Initial microservice releases lacked monitoring, CI/CD, and data‑statistics tools. With support from a two‑person operations team, the group built a unified monitoring, alerting, and deployment pipeline, enabling rapid rollout of subsequent services.

Current split plan :

Legacy services remain for backward compatibility (Bubble 1.0).

New independent services (HTTP for App, RPC for internal calls) cover circles, users, group chat, etc.

Common components have been turned into shared microservices (e.g., user entity, circle entity, event topics).

Front‑end of the operation backend has been migrated from JSP to Vue for better separation.

Company‑wide services such as comments, likes, voting, and points have been re‑bounded to clear system boundaries.

Challenges encountered : Deciding top‑down vs. bottom‑up decomposition, limited developer bandwidth due to bi‑weekly releases, and early RPC instability (client exceptions, full GC) that required connection‑pool tuning and server‑side memory optimization.

Future directions : Implement full‑link tracing, comprehensive load‑testing, CI integration for testing and monitoring, global transaction consistency, and unified task scheduling. The team aims to continuously iterate as business requirements evolve.

Conclusion : Microservices are not a panacea; they must serve business growth. The Bubble system’s evolution demonstrates that careful, business‑driven decomposition, performance‑aware technology choices, and strong ownership can sustain a platform with tens of millions of daily users.