From Offline to Real-Time Recommendation: iQIYI’s Scalable Machine Learning Journey

In iQIYI’s ecosystem, “users” and “content” are core concepts. The recommendation team leverages machine‑learning techniques to match users with content they may like, and to find users who may be interested in a given piece of content. Behind this simple principle lies complex algorithmic computation and massive data processing.

The team’s story begins with offline recommendation frameworks, which were much slower than the emerging online feed‑based recommendation. To keep up with the high QPS (queries‑per‑second) and frequent updates, the online service initially suffered a 4% timeout rate during peak periods.

Through systematic performance analysis, the team applied multiple optimizations: data pre‑processing at the source, database sharding, master‑slave replication, secondary caching, and user‑ID hash partitioning. These measures reduced the peak‑time timeout rate to below 0.3%.

Initially only three engineers worked on online recommendation, but after several iterations the recommendation frequency was reduced from hourly to seconds, eventually achieving real‑time online recommendation.

Real‑time recommendation enables users to refresh the most interesting video content at any moment, and each user action immediately influences the next recommendation.

The underlying infrastructure now collects, cleans, and stores massive data streams in real time. User video interactions are analyzed instantly, and richer user tags are generated to capture preferences, allowing real‑time matching between users and content.

To support this, the data platform evolved from offline manual processing to an automated online data middle‑platform, transitioning from traditional MapReduce to real‑time technologies such as Kafka, Spark‑Streaming, and Flink. The system can now handle tens of thousands of events per second, continuously updating user interests and delivering more precise recommendations.

Beyond personalization, the team emphasizes diversity to avoid over‑exposure to a single content type, promoting a richer user experience. They also use recommendation algorithms to strengthen the connection between creators (UP hosts) and fans, boosting private traffic and community interaction.

Only high‑quality, positive‑energy content is prioritized, reflecting iQIYI’s vision of “greatness” in algorithmic decisions.

From a management perspective, the speaker shares two key insights: always ask “why” before acting to clarify goals, and ensure technical leaders stay close to the front‑line engineering work to keep technology up‑to‑date and avoid purely abstract management.

Over six years, iQIYI has progressed from a follower to a leader in recommendation technology, a source of pride for the team.