Design and Implementation of iQIYI CDN Inspection System

CDN (Content Delivery Network) is a distributed network system that uses edge servers deployed worldwide to deliver content close to users, reducing congestion and improving response speed. iQIYI, a leading video entertainment provider in China, relies on a stable CDN to ensure high‑quality, smooth video experiences for over 500 million monthly active users.

iQIYI operates a hybrid CDN architecture that combines self‑built CDN nodes with commercial CDN services, totaling tens of thousands of servers. While this hybrid model improves resource utilization and reduces cost, it also introduces challenges in software/service change management and quality monitoring. Configuration inconsistencies, incomplete files, high hijack rates, and non‑local traffic routing have caused service instability.

To proactively monitor the health of all CDN servers and detect issues before deployment, iQIYI designed and implemented a CDN Inspection System consisting of three main subsystems:

1. Inspection Task Generation System : Generates customized inspection cases for millions of CDN servers, supporting manual or scheduled task creation. It includes an inspection case selector, a real‑time IP database, and an inspection case generator that writes tasks to Redis.

2. Central Processing System : Acts as the system’s “brain”, splitting inspection tasks, distributing them via a configuration and task dispatch system, collecting results from edge measurement agents, and performing real‑time analysis. It integrates a data processing center (using Celery and ZeroMQ for high‑throughput task splitting and result collection), a configuration & task dispatch system (supporting minute‑level synchronization for tens of thousands of servers), and an intelligent scheduling system that triggers automated remediation actions based on fault type.

3. Edge Measurement System : Deployed across multiple ISPs and provinces, this subsystem executes the actual measurements. Agents run non‑intrusive scripts and support two measurement modes: IP mode (direct IP probing for functional tests) and domain‑resolution mode (simulating real user access to assess local coverage, hijacking, and directed traffic routing).

The system addresses two major technical challenges: ensuring real‑time, accurate CDN service IP information (with 5‑minute API updates and multi‑source IP validation) and maintaining an extremely low inspection case generation error rate (<0.001%) despite diverse CDN access specifications.

Application scenarios include pre‑deployment checks for service or configuration changes, continuous monitoring of both self‑built and commercial CDN nodes, and automated fault handling (e.g., automatic up/down of self‑built CDN nodes, coordinated remediation with commercial CDN providers).

After six months of stable operation, the inspection system has detected over 100 functional/configuration issues, 10 file MD5 mismatches, and 5 directed‑traffic anomalies, prompting corrective actions from three commercial CDN providers and achieving 100% local coverage. Future work will expand inspection modes and incorporate AI techniques to further safeguard iQIYI’s IOCP and Qisu streaming services.