Designing a High‑Throughput Log Collection System with ClickHouse, UDP and Compression

In daily work, logs such as request parameters, info and error messages need to be stored for troubleshooting; common solutions include local files, ELK, Kafka, and filebeat, but they become costly at large scale.

When log volume grows to gigabytes per second, traditional pipelines (local files → MQ → ES) require thousands of servers, massive disk I/O, and high network bandwidth, making the solution unsustainable.

The proposed redesign removes unnecessary steps: logs are written to memory, compressed (Snappy/ZSTD), and sent directly via UDP to a worker cluster, bypassing local disk and MQ.

Workers buffer data in a double‑queue, parse and batch‑write to ClickHouse, a column‑oriented OLAP database with high compression and write performance; queries are served through a dashboard.

Key components: a configuration center for worker IPs, a client SDK that captures request parameters and logs via custom appender, UDP transport with fallback to HTTP for oversized packets, and workers that use large‑memory containers and multithreaded consumption.

Performance tests show a single worker can handle 1‑5 × 10⁷ raw log rows per second, processing ~2 × 10⁴ QPS, and write 160‑200 MB/s to ClickHouse; the new system reduces disk usage to <80% of the original and cuts overall hardware cost by more than 70%.

ClickHouse optimizations include a high‑availability proxy layer, direct writes to local tables (avoiding distributed‑table overhead), and efficient partitioning and indexing for fast multi‑dimensional queries.

The final comparison demonstrates that the new architecture dramatically lowers storage and CPU costs while delivering ten‑fold higher consumption performance compared to the previous MQ‑based pipeline.