How Pupu APM Boosted Write Speed and Cut Storage Costs with ClickHouse

Background

Rapid growth of the e‑commerce platform increased the volume of spans reported to the APM system. Elasticsearch (ES) clusters experienced high write latency and frequent slow queries, forcing the operations team to upscale the clusters and raise costs.

Why ClickHouse?

ClickHouse (CK) was selected for its higher write throughput, better compression, and native materialized view support. In benchmark tests ES timed out when ingesting 1.8 million spans per second, while CK sustained 3 million spans per second. With GZIP compression CK stored 10 billion spans in 2.8 TB versus 4.5 TB for ES—a 38 % reduction. Materialized views enabled pre‑aggregation of metrics over a seven‑day window, a capability not available in ES.

ClickHouse Storage Practice

Application‑layer Read/Write Tuning

Zero‑Downtime Data Source Switch

Dual‑write to ES and CK for seven days.

A reconciliation process verified that row counts and sampled data matched between the two stores.

After seven days traffic was switched to CK; if issues arose, a fallback to ES was kept for another seven days.

ES was decommissioned after two weeks of stable operation.

Optimizing CK Write Rate

Used TabSeparated format instead of VALUES. Inserting 250 k rows per batch took 4 seconds with TabSeparated versus 15 seconds with VALUES.

Introduced an in‑memory queue to increase concurrent inserts. The queue workflow is illustrated in the image below.

Figure 1: In‑memory queue workflow

Online Deployment Tuning

CK Deployment Options

The online CK cluster consists of data nodes and coordinator nodes. Hot data (last 8 hours) is stored on high‑performance EBS; older data is stored on cheaper S3, providing a seven‑day retention period. A redundant JSON column was added to CK tables to store user‑defined fields, enabling flexible queries.

Figure 2: Online CK cluster architecture

Cluster Issues and Solutions

Issue 1: An initial 5‑node cluster (32 CPU/256 GB each) consumed only 1.65 million spans/s, far below the 3 million target.

Solution: Added three 48‑CPU/384 GB coordinator nodes and increased thread settings (parts 300 → 10 000, max_thread 1 → 16, thread pool 16 → 64).

Issue 2: After scaling, throughput rose to 1.8 million spans/s but plateaued; logs showed excessive parts queues despite low CPU, disk, and network utilization.

Solution: Analyzed data characteristics, separated production and consumption queues (distinct TraceId per span), filtered unreasonable data, and reduced parts‑queue pressure, achieving 2.4 million spans/s.

Issue 3: Network bandwidth became the bottleneck while CPU and memory remained low.

Solution: Downgraded coordinator nodes to 32 CPU/64 GB (3 nodes) and added two more data nodes (now 7 × 32 CPU/64 GB). Peak throughput increased to 3 million spans/s without raising overall cost; data‑node CPU averaged 50 %.

Cost and Performance Comparison

After the optimizations, the CK cluster was compared with the original ES cluster.

Cost: CK reduced storage cost by 2.1 W/month relative to ES.

Performance: Write rate nearly doubled while unit cost decreased.

Conclusion and Outlook

The migration from Elasticsearch to ClickHouse increased write throughput to the target 3 million spans per second, eliminated ingestion latency, and significantly lowered machine costs. Future work will explore head‑ and tail‑sampling techniques to further improve APM performance and reduce operational expenses.