Building Prism: Qunar’s Real‑Time Data Platform and DevOps Journey

Qunar’s online real‑time data platform, named Prism, was created to address the difficulty of locating problems in production by building a visual, low‑cost data analysis system that integrates log collection, streaming, and storage components.

Prism provides services such as real‑time log collection with ELK, a high‑throughput data bus based on Kafka for inter‑departmental message distribution, and big‑data streaming analysis using Spark Streaming, Flink, or Storm, as well as Elasticsearch‑as‑a‑Service for storage and Zeppelin+Spark/Flink for OLAP and experimentation.

The data flow starts with a collection layer (Rsyslog, QFlume, Heka, Packetbeat, etc.) feeding into Kafka, where ETL processing via Logstash transforms logs before indexing them into Elasticsearch for Kibana visualization, serving both engineers and product managers for monitoring and troubleshooting.

To overcome the bottleneck of manual bug investigation, the team adopted a DevOps approach, centralizing log access and enabling rapid issue detection through ELK, which dramatically reduced deployment windows from six‑seven hours overnight to about one‑two hours.

As requirements grew, Prism’s architecture evolved to incorporate Docker, Marathon, and Mesos, turning fixed resources into elastic, centrally managed pools, simplifying deployment, and allowing dynamic scaling of services.

Higher‑level data analysis was achieved by integrating a Spark scheduler on Mesos (later migrated to Marathon), enabling cluster‑wide Spark jobs and supporting real‑time joins, recommendations, and other advanced analytics.

In summary, Prism delivers a real‑time monitoring and analysis platform that lowers the barrier to data‑software deployment, improves operational efficiency, and provides a foundation for future enhancements such as GPU‑accelerated neural network workloads.