Mesos‑Based Log Platform Architecture and Practices at Qunar DevOps

Platform Introduction

In May of this year the team began researching and trialing Mesos, with the first pilot being a log platform that runs all log analysis on Mesos. The platform serves developers, testers, and operators across business lines, enabling quick issue location, traceability, and operational reporting.

The platform uses the ELK stack (Elasticsearch, Logstash, Kibana) for log analysis, with minor customizations to Kibana and Logstash to integrate with the company’s monitoring and authentication systems. Logs are collected via rsyslog (kernel, mail, cron, dmesg, etc.), Flume for business logs, and Mozilla Heka and Fluentd for container logs.

Mesos Technology Stack

The team employs two Mesos frameworks: Marathon and Chronos, plus a custom monitoring framework called Universe.

Marathon’s eventSubscriptions feature enables powerful HTTP callbacks, which are used for container monitoring. Setting minimumHealthCapacity reduces resource consumption during restarts. Service discovery is handled via a custom OpenResty plugin that dynamically loads Marathon application info and proxies external requests to the appropriate Mesos service.

Non‑HTTP services (e.g., statsd UDP) use Mesos DNS with fixed ports, while dynamic‑port services rely on entrypoint‑generated hostnames. For applications with the UNIQUE attribute, the team wrote a daemon that watches for new machines and automatically scales the UNIQUE instances.

To address resource fragmentation, a custom Mesos framework calculates real resource leftovers and simulates deployments, feeding the data to the monitoring/alerting system. Chronos runs scheduled cleanup and monitoring scripts.

Docker runs in host network mode; container monitoring uses cAdvisor and Heka. A Docker SSH proxy (implemented with docker exec -it ) allows developers to SSH into containers directly from their machines.

Application Solution

The log platform processes over 5.1 billion logs daily across more than 600 containers, maintaining latency between 60–100 ms. Images are treated as runtime environments rather than code repositories; configuration files are version‑controlled in GitLab and pulled at container start‑up based on image tags.

Logstash was extended with custom Ruby code to add monitoring of two internal queues, EPS, and parsing latency. Kafka partition lag statistics run on Chronos, and dedicated Logstash instances handle traffic from each data center.

Container monitoring initially used a custom framework that queried Mesos APIs, but later switched to cAdvisor for UNIQUE applications. Data is aggregated and sent to the company’s monitoring platform via StatsD.

Deployment is integrated with Jenkins: developers push filters to GitLab, tag the repository, and Jenkins builds and publishes the image. The system automatically creates Kibana indices and dashboards, recycles unused Kibana containers, and supports sandbox environments for local debugging.

The team is also experimenting with Storm on Mesos, though they note current limitations.

Q&A

Q: How does Mesos differ from YARN? A: Mesos provides a two‑level scheduler that allocates resources to multiple frameworks (e.g., Hadoop, Storm, Spark) allowing fine‑grained sharing, whereas YARN uses a single central scheduler with limited flexibility for framework‑specific algorithms. Q: How is data persistence handled? A: Persistence is kept outside the cluster; Mesos 0.23 offers persistence support but it has not been used in production. Q: Is Storm on Mesos production‑ready? A: The code is available on GitHub but lacks many resource‑control features and exhibits slow message acknowledgments; it is not recommended for production. Q: How are images built and stored? A: Images are built by Jenkins and stored in Swift. Q: Why use host networking for Docker? A: Host mode provides higher throughput for the high‑volume platform; bridge mode performance was insufficient. Q: How does the scheduling algorithm work and is high availability provided? A: Mesos offers a two‑level offer‑based scheduler; Marathon already provides high availability. Q: What is the scale and resource utilization? A: About 600 containers run on roughly 30 mixed VMs and bare‑metal machines, with ample remaining resources. Q: Why choose Heka over Fluentd and what issues exist with Logstash? A: Both Heka and Fluentd collect container logs, but Heka can read more environment variables; Logstash was extended for better monitoring and lower resource usage.

Source: http://dockone.io/article/675