Real-Time Log Monitoring and Alerting System for iQIYI Membership Services

In June 2019 iQIYI's membership reached 100 million users, prompting rapid growth of its machine clusters and exposing limitations of the existing monitoring system.

The new monitoring solution collects four types of logs (Nginx, application, exception, front‑end) in real time, aggregates them, and provides minute‑level alerts.

To handle billions of log events, the architecture adopts Spark Streaming and Flink for stream processing and Druid as the real‑time analytical datastore, with Kafka as the transport layer.

Key components include Venus‑Agent for log collection, Kafka for ingestion, real‑time processing pipelines, Druid for storage and OLAP, and offline jobs for long‑term analysis.

The system offers multi‑dimensional monitoring, automated alerting via thresholds and spikes, and supports operational actions such as degradation, traffic switching, and rate limiting.

Challenges addressed comprise log standardization across VM and QAE deployments, collector performance bottlenecks, resource cost optimization, and latency in Spark/Flink consumption.

Future work focuses on intelligent threshold tuning, traffic‑prediction models, and enhanced automated fault localization.