Why Modern Systems Need Log Analysis Platforms – Ctrip’s ELK Case Study

Why Use a Log Analysis Platform

Logs are crucial for any website or system, whether small or large. In the early stages, administrators often manually inspect web server access logs to monitor 404 errors or identify performance bottlenecks. As traffic increases, the sheer volume of logs makes manual inspection impractical.

When a single server can no longer handle the load, administrators must move to clusters, and analyzing logs across many machines becomes impossible without tools. Traditional methods such as grep, sed, and awk are insufficient for large‑scale environments.

Log analysis serves several key purposes:

Monitoring system health

Diagnosing the root cause of bugs

Tracing security incidents

Providing data for website planning and activities

Because log volume grows rapidly and sources become distributed, storing logs in plain files is no longer adequate; many organizations now persist logs in databases such as MySQL, MongoDB, or HBase.

These trends make traditional log‑analysis techniques ineffective, creating a need for dedicated log analysis platforms.

Ctrip Case Study

Ctrip, China’s largest OTA website, generates dozens of log types amounting to several terabytes daily. Using commercial software like Splunk would cost tens of millions of yuan per year, so Ctrip built its own platform.

Earlier tools based on MySQL and HBase were limited to simple viewing and filtering; complex queries and statistics were slow and provided a poor experience. The operations team defined the following requirements for a new log analysis tool:

Support multiple data sources

Flexible and simple log parsing

Keyword search and browsing with combinable conditions

Time‑window based statistical calculations on specific fields (e.g., average response time, most frequent error URLs)

After evaluating options, Ctrip adopted the ELK stack and extended it for their needs.

ELK Platform Overview

ELK consists of three open‑source components:

Elasticsearch : a distributed search engine that provides fast search and aggregation over massive log data. It features automatic node discovery, index sharding, a RESTful API, multi‑source ingestion, and load‑balanced searching.

Logstash : a fully open‑source pipeline for collecting, parsing, and forwarding logs to storage.

Kibana : an open‑source visualization tool that works with Elasticsearch to present aggregated log data in an intuitive, user‑friendly interface.

The typical workflow is:

Deploy Logstash agents on servers that need to collect logs.

Each agent forwards filtered logs to a Logstash indexer , which aggregates them.

The aggregated logs are stored in Elasticsearch , where administrators can configure custom search queries.

Finally, Kibana visualizes the search results and dashboards for analysis.

By leveraging ELK, Ctrip achieved a scalable, flexible, and cost‑effective log analysis solution that meets the operational demands of a massive, high‑traffic platform.