Design and Evolution of iQIYI Real-Time Analysis Platform (RAP)

In the era of information explosion, the need for large‑scale data analysis, real‑time mining, and data‑driven decision making has become critical. iQIYI built a minute‑level latency real‑time analysis platform, RAP (Realtime Analysis Platform), based on Apache Druid combined with Spark/Flink, to provide fast, accurate, and agile OLAP capabilities.

RAP enables users to configure massive real‑time multidimensional analysis through a web wizard, automatically generating OLAP models and minute‑level visual reports. It also offers APIs for aggregated data retrieval and integrates with various business lines such as membership, recommendation, and BI, supporting hundreds of streaming tasks and thousands of analytical reports.

1. Real‑Time Analysis Requirements

Since 2010, iQIYI’s data platform evolved from a Hive+MySQL offline OLAP warehouse to a hybrid architecture involving Kylin, Impala, Kudu, Druid, and Elasticsearch. The legacy offline warehouse could no longer meet the low‑latency demands of the business, leading to the development of RAP.

Before RAP, building a real‑time analysis service faced four major challenges:

Difficulty in selecting the right OLAP engine.

High development cost (Spark/Flink coding + front‑end report development).

Poor data freshness (latency from minutes to days).

Heavy maintenance when data sources changed.

RAP addresses these issues by offering a web‑guided configuration that creates large‑scale real‑time multidimensional analysis and minute‑level visual reports while reducing development and maintenance effort.

2. Architecture Evolution

2.1 RAP 1.x

The first generation chose Druid as the underlying OLAP engine because it provides sub‑second query latency for time‑series data, making it ideal for real‑time aggregation.

2.2 RAP 2.x

RAP 2.x introduced several enhancements:

Integration of Kafka Indexing Service (KIS) for exactly‑once ingestion, supporting Druid versions ≥0.10.x and advanced features such as HLL Sketch for distinct counting.

Support for Apache Flink as an alternative stream processing engine, reducing processing latency from minutes to seconds and preserving exactly‑once semantics.

Enhanced task diagnostics, including stream‑task latency charts, real‑time ingestion monitoring, and error data sampling, enabling rapid root‑cause analysis.

3. Business Applications

3.1 Membership Log Monitoring

In June 2019, iQIYI’s membership exceeded 100 million users. RAP processes hundreds of billions of log records daily, delivering minute‑level alerts and generating over 700 real‑time reports, improving fault‑resolution speed by 80 %.

3.2 Recommendation Algorithm Effectiveness

RAP monitors click‑through rates (UV/VV) and viewing time (PPUI) for recommendation algorithms, enabling algorithm switches within 30 minutes and accelerating iteration cycles.

3.3 Smart TV Real‑Time Alerting

RAP analyzes playback errors across dimensions such as client version, server IP, and city, providing a 5‑minute alert loop that helps quickly locate and resolve streaming issues.

4. Future Directions

Future work will focus on finer‑grained monitoring of the analysis pipeline, richer diagnostics, higher resource utilization, and stronger exactly‑once guarantees, as well as extending capabilities to retention analysis and intelligent analytics.