Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

This article details Bilibili's evolution of its log system from an Elastic Stack‑based solution to a ClickHouse‑backed architecture with OpenTelemetry, describing the challenges of cost, stability, and scalability, the new components such as Log‑Agent, Log‑Ingester, and a custom visualization platform, and the performance gains and future directions.

OpenTelemetryclickhouselarge-scale logging

0 likes · 26 min read

Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

Big Data Technology Architecture

Sep 17, 2022 · Databases

Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

This article describes how Bilibili redesigned its log service by replacing Elasticsearch with ClickHouse, introducing OpenTelemetry‑based logging, optimizing storage, query, and alerting components, and enhancing ClickHouse features such as configuration tuning, Map types, and implicit columns to achieve higher performance, lower cost, and better observability.

Database OptimizationOpenTelemetryclickhouse

0 likes · 28 min read

ITPUB

Sep 16, 2022 · Big Data

How Bilibili Re‑engineered Its Log Service with ClickHouse and OpenTelemetry for 10× Performance

Bilibili redesigned its five‑year‑old ELK‑based log platform by replacing Elasticsearch with ClickHouse, adopting OpenTelemetry for unified log ingestion, and building a custom visualization and alerting system, achieving tenfold write throughput, one‑third storage cost, and dramatically faster query response times.

OpenTelemetryclickhouselog infrastructure

0 likes · 28 min read

How Bilibili Re‑engineered Its Log Service with ClickHouse and OpenTelemetry for 10× Performance

Bilibili Tech

Sep 16, 2022 · Big Data

Design and Optimization of Bilibili Log Service 2.0 Using ClickHouse and OpenTelemetry

Bilibili’s Log Service 2.0 replaces its Elastic‑Stack pipeline with an OpenTelemetry‑driven architecture that writes logs via high‑performance Go/Java SDKs to ClickHouse, delivering ten‑fold write throughput, two‑fold query speed, one‑third storage cost, a custom query gateway, visualization UI, and advanced alerting.

OpenTelemetryPerformance Optimizationclickhouse

0 likes · 27 min read