How Vipshop Replaced ELK with ClickHouse for a Scalable, Low‑Cost Log System

Background

Vipshop's Dragonfly log system, built on an EFK stack, has been in production since 2014, handling physical‑machine and container logs. Business growth caused a surge in log volume, longer retention requirements, and higher hardware/maintenance costs for the 260‑node Elasticsearch cluster.

Log System Evolution

1. Standard Log Format

A unified JSON format was defined, containing timestamp, level, thread name, class name, eventName and custom fields. Filebeat parses this format and forwards it.

From the standardized logs, Filebeat produces JSON records as shown below.

Supports various aggregation scenarios during query.

Custom fields enable alerting with aggregation functions.

2. Problems with the ES Storage Model

Each domain creates a daily index; large indices cause heavy CPU for segment merging, and the 5.5 ES version lacks automatic index deletion and hot‑cold migration. Scripts for index lifecycle are long‑running and fragile, and the inverted index consumes a lot of disk.

Advantages of ES: fast query per shard, high QPS.

In massive log scenarios the full‑text index becomes wasteful because only a tiny fraction of logs are actually queried.

3. 2.0 Architecture – ClickHouse‑Based Solution

After evaluating an HDFS‑based approach, the team chose ClickHouse for its columnar storage, high compression (lz4, zstd) and vectorized execution. Benchmarks showed zstd achieving up to 15.8× compression and query speed comparable or superior to ES.

Key components:

Log ingestion – vfilebeat : custom Go agent that tags logs with domain and host, reads container POD info, and pushes JSON to Kafka.

Log parsing – Flink writer : consumes Kafka, drops expired logs (>3 days), filters malformed entries, maps fields to ClickHouse schema, and batches data into per‑domain queues.

Storage – ClickHouse MergeTree : uses partition key, primary key, and skip‑indexes (minmax, set, bloomfilter) for fast pruning; part size limited to 30 GB, background merges reduce part count.

TTL management : table‑level TTL replaced by partition‑level TTL with a daily cleanup job, allowing mixed retention in a single table.

Custom field storage : arrays of *_names and *_values per data type; queries use array indexing functions. An alternative map‑based storage was tested but proved slower.

Technical Details

vfilebeat runs on the host, can be configured per domain, and supports both physical‑machine and container logs. Flink writer converts logs to ClickHouse rows, optimizes JDBC batch creation to reduce memory by ~33 % and CPU by ~13 % by replacing map‑based escaping with a switch.

ClickHouse’s MergeTree stores data in parts, each part sorted by the primary key; indexes are tiny and kept in memory, enabling high‑throughput writes when part count, CPU cores, and disk I/O are balanced.

High‑speed query tricks include using time, tag, and bloom‑filter pruning before vectorized execution; the engine’s SIMD‑friendly design reduces cache misses and yields several‑fold speedups over non‑vectorized scans.

Frontend Log Query System

The new UI automatically injects domain and time range into user queries, supports multi‑tenant isolation, and simplifies custom‑field syntax (e.g., str.name instead of string_values[indexOf(string_names,'name')]).

ClickHouse executes a single statement per query; to avoid double‑scanning for histograms and TOP results, the system narrows the time window based on histogram buckets before the TOP query.

Best Practices

Keep individual log messages under 10 KB.

Always include indexed tags in query conditions to reduce result set size and improve speed.

Leverage ClickHouse’s OLAP strengths for massive log storage and analysis with low cost, high availability, and high‑throughput queries.