How Bilibili Re‑engineered Its Log Service with ClickHouse and OpenTelemetry for 10× Performance

Background

Log data is essential for online troubleshooting and observability, requiring stability, cost‑effectiveness, ease of use, and scalability. Bilibili's ELK‑based log system (Billions) has been in production since 2017, now spanning over 500 machines and ingesting more than 700 TB of logs daily.

Problems Encountered

High storage cost and CPU‑intensive tokenization in Elasticsearch caused bottlenecks in write throughput and real‑time latency.

Limited compression led to excessive memory usage, forcing frequent sampling and rate‑limiting.

Warm‑phase indices needed manual closing, reducing usability.

Dynamic mapping had to be disabled, complicating user queries.

Lifecycle management before ES 7 required custom components, increasing maintenance overhead.

Kibana’s complex codebase made extensions hard and upgrades costly.

Internal JSON‑based log format SDKs for Java and Go suffered from average serialization performance and compatibility issues.

New Architecture Overview

The redesigned Bilibili Log Service 2.0 replaces Elasticsearch with ClickHouse for storage, introduces a self‑built visual analysis platform, and adopts OpenTelemetry as a unified log reporting protocol.

Core Components

OTEL Logging SDK : High‑performance structured‑log SDK for Go and Java implementing the OpenTelemetry logging model.

Log‑Agent : Deployed on physical hosts, receives OTEL logs via domain socket, performs low‑latency file collection, and supports container environments.

Log‑Ingester : Subscribes to Kafka, partitions logs by time and metadata, and batches writes into ClickHouse.

ClickHouse : Columnar storage with high compression, implicit columns for dynamic schema, and superior query speed.

Log‑Query : Handles routing, load‑balancing, caching, rate‑limiting, and simplifies query syntax.

BLS‑Discovery : In‑house visual analysis platform offering Kibana‑like UI with zero learning curve.

ClickHouse‑Based Log Storage

Switching to ClickHouse yielded a ten‑fold increase in write throughput and reduced storage cost to one‑third of the previous system. Structured field queries became twice as fast, with 99 % of queries completing within three seconds.

Write performance surpassed Elasticsearch by more than ten times.

Dynamic schema is supported via implicit columns (string_map, number_map, bool_map) that store common log fields while keeping query performance high.

TTL policies define Hot (24 h, high‑performance storage), Warm (SATA, searchable), and Cold (deleted or HDFS‑backed) stages, with ZSTD(1) compression improving space efficiency by 50 % without noticeable performance loss.

Query Gateway

The gateway abstracts underlying storage details, providing routing, load‑balancing, simplified SQL‑like syntax, caching, and rate‑limiting. Users can write simple SQL without worrying about ClickHouse’s local/distributed tables or implicit columns, and a Luence‑to‑SQL parser further eases migration.

Self‑Built Visual Analysis Platform

Provides a Kibana‑like UI to minimize migration effort.

Integrates with monitoring, distributed tracing, and alerting.

Features query highlighting, field distribution analysis, time‑distribution preview, and log snippet previews.

Code‑mirror2 powers auto‑completion and suggestions, reducing the learning curve for new users.

Log Alerting

Alert rules now use ClickHouse as the data source, with a unified calculation model that removes Elasticsearch‑specific semantics. A rule consists of name, data source, time range, compute interval, function (count, sum, max, distinct count, etc.), filter expression, trigger condition, channel, message template, notification group, and storm‑suppression settings.

Over 5,000 alert rules are already deployed, with a migration tool that converts legacy ES rules to the new format.

OpenTelemetry Logging

OpenTelemetry, the unified observability framework, defines stable APIs for logs, metrics, and traces. Bilibili implemented Go and Java SDKs conforming to the OTEL logging model and integrated the OTEL‑compatible layer into Log‑Agent.

Improving Log Search

For small log volumes, ClickHouse can search directly. For large volumes, a secondary tokenbf_v1 index on map keys and the use of the ~` operator enable near‑ES performance. Structured logging is encouraged to improve searchability. log.Info("report id=32 created by user 4253") After structuring:

log.Infov(log.KVString("log_type", "report_created"), log.KVInt("report_id", 32), log.KVInt("user_id", 4253))

ClickHouse Optimizations

Configuration Tuning

Adjusted min_bytes_for_wide_part and min_rows_for_wide_part to favor compact parts, reducing part count.

Increased max_bytes_to_merge_at_min_space_in_pool and max_bytes_to_merge_at_max_space_in_pool to allow larger merges under load.

Expanded background_pool_size from 16 to 32 threads for more merge capacity.

Zookeeper Load Mitigation

Introduced auxiliary Zookeeper clusters to distribute metadata writes, alleviating the bottleneck caused by many tables/partitions.

Map Type Enhancements

Native ClickHouse Map lacks indexing and forces full deserialization. Bilibili added a tokenbf_v1 index on map keys and later created an implicit‑column implementation (MapV2) that stores each map key as a separate column, enabling index push‑down and eliminating unnecessary I/O.

Creating a test table:

CREATE TABLE bloom_filter_map (</code><code>    `id` UInt32,</code><code>    `map` Map(String, String),</code><code>    INDEX map_index map TYPE tokenbf_v1(128, 3, 0) GRANULARITY 1</code><code>) ENGINE = MergeTree ORDER BY id SETTINGS index_granularity = 2;</code><code>INSERT INTO bloom_filter_map VALUES (1, {'k1':'v1','k2':'v2'});</code><code>INSERT INTO bloom_filter_map VALUES (2, {'k1':'v1_2','k3':'v3'});</code><code>INSERT INTO bloom_filter_map VALUES (3, {'k4':'v4','k5':'v5'});</code><code>SELECT map['key1'] FROM bloom_filter_map;

Implicit‑column tests showed significant speedups for key‑specific queries.

Limitations

Cannot select the entire map field because it is split into multiple columns.

Sparse key distributions can cause an explosion of implicit columns; a max_implicit_columns parameter caps the number and returns an error when exceeded.

Future Work

Develop log‑mode extraction to handle unstructured text for compression, post‑processing, and anomaly detection.

Integrate lake‑warehouse (lakehouse) concepts, allowing long‑term low‑cost storage of compliance logs and enabling downstream ML/BI use cases.

Explore direct agent‑to‑ClickHouse ingestion, removing Kafka, and tighter ClickHouse‑Iceberg integration.

Improve ClickHouse full‑text search to close the gap with Elasticsearch.

References

https://opentracing.io/

https://opencensus.io/

https://opentelemetry.io/

https://opentelemetry.io/docs/collector/

https://github.com/ClickHouse/ClickHouse/pull/28511