iQIYI Magic Mirror: Evolution of a Big Data Analysis Platform

The Magic Mirror platform was created to solve the bottleneck caused by data engineers when business units required rapid, diverse data reports in a fast‑moving internet environment. Fixed reporting could not keep up, so the platform empowers users to obtain data themselves.

Development progressed through three major stages: Magic Mirror 1.0 (2015) supported basic calculations and Hive execution; Magic Mirror 2.0 (2019) added data‑warehouse table registration, templated calculations (basic, join, retention), and a distributed Gear workflow to replace single‑point Hive execution; Magic Mirror 3.0 (2022) introduced a unified storage layer (Hive, Iceberg, local), a pilot engine that routes queries to Spark SQL, Hive, Trino, or Impala, and richer visual dashboards.

Each version improved functionality: 1.0 offered simple three‑step configuration for basic calculations; 2.0 introduced multi‑table join templates, retention analysis, and custom SQL editing, boosting daily active users by 25% but still relied on Hive, limiting latency and visualization. 3.0 added a pilot engine with syntax parsing, query interception, intelligent routing, and multi‑engine support, cutting average task time from ~20 minutes to ~6 minutes (≈70% reduction).

The data layer in 3.0 consolidates raw logs, business data, and other sources into an ODS, then processes them into DWD, MID, and DIM layers, supporting both unified warehouse and data‑mart models. This architecture provides real‑time ingestion, standardized metrics, and high data quality, enabling efficient multi‑table analysis without user awareness of underlying tables.

Custom SQL in 3.0 supports scenario‑based analysis and built‑in chart visualizations (trend, pie, bar, etc.), eliminating the need to export data for external tools. The platform also defines a unified metric metadata system, allowing business users to select metrics and dimensions that automatically map to optimal physical tables, improving query performance.

Benefits observed after launch include full‑business coverage across iQIYI, a shift from day‑level to minute‑level data freshness, cost reduction by decommissioning hundreds of query servers, and enhanced data security through SQL monitoring and write‑operation interception.

Future plans focus on expanding query‑engine support and increasing analysis intelligence, moving beyond configuration‑heavy templates toward more automated, AI‑assisted analytics.