iQIYI Technical Product Team
Sep 15, 2023 · Big Data
Apache Spark at iQIYI: Current Status and Optimization
iQIYI now relies on Apache Spark as its main offline engine, processing over 200 000 daily tasks for ETL, data synchronization and analytics, while recent optimizations—dynamic resource allocation, adaptive query execution, compression, rebalance, Z‑order and resource‑governance—have cut compute usage by ~27 %, storage by up to 76 % and improved query speed, completing a large‑scale migration from Hive and paving the way for Spark 3.4 and Iceberg support.
Apache SparkBig DataSQL Service
0 likes · 21 min read