How Bilibili Built Billions 3.0: A Low‑Cost, Scalable Log Platform with ClickHouse, Iceberg, and Trino

Background and Key Challenges

After deploying the ClickHouse‑based Billions 2.0 log solution, Bilibili achieved a 60% reduction in storage cost but still faced five major issues: further storage‑cost optimization, limited log‑troubleshooting capability, the constraints of a shared‑nothing ClickHouse architecture for massive logs, the need for complex ETL workflows, and low overall resource utilization.

Industry Research

The team evaluated OpenSearch, ClickHouse, Loki, SLS and internal solutions, categorizing them into two camps: storage‑compute‑integrated (share‑nothing) and storage‑compute‑separated architectures. OpenSearch and ClickHouse offered strong performance but required significant engineering effort; Loki was lightweight but suffered from small‑file overhead; ultimately, none matched Bilibili's specific constraints.

Billions 3.0 Architecture Overview

The new design follows a lake‑house model that combines a low‑cost, scalable storage layer (Iceberg tables on HDFS) with an acceleration layer (ClickHouse) and a query engine (Trino). The architecture remains open‑ended, allowing future component swaps.

Core Components

Log‑Agent : OTEL‑compatible collector deployed as a daemon on physical machines and containers; supports dozens of log formats, parsing, filtering, sampling, and pushes structured logs downstream.

Log‑Gateway (Kafka‑Proxy) : Aggregates logs from agents and routes them to dedicated Kafka clusters (high‑priority, dedicated, general). Provides transparent pipeline degradation and load‑balancing.

Data Pipeline : Kafka clusters act as the backbone for traffic shaping and back‑pressure handling.

Log‑Consumer : Golang‑based, containerized consumer that performs lightweight processing; for complex cases it delegates to Flink jobs.

Log Engine : Combines ClickHouse (for hot data) and Iceberg + Trino (for cold data) to provide both fast point‑queries and scalable analytical queries.

Query Gateway : Uniform DSL/SQL façade that hides the underlying engine differences; integrates with Grafana for metric dashboards.

Storage Layer Choices

Iceberg was selected as the table format because it is widely adopted, schema‑evolution friendly, and supports both batch and streaming workloads. HDFS was chosen as the underlying storage due to its massive capacity, native integration with the broader data lake ecosystem, and EC 6+3 erasure coding that yields ~1.5× storage overhead versus ClickHouse’s double‑replica model.

Table Schema Design

Each business creates its own Iceberg table partitioned by day, with optional secondary partitions on high‑cardinality business fields. Columns are divided into:

Public fields : timestamp, app_id, etc., present in every log.

log_msg : raw text for full‑text search.

Private fields : business‑specific dimensions stored as MAP or JSON, allowing dynamic schema evolution.

Indexing Strategies

To accelerate queries, the platform implements both forward and inverted indexes on Iceberg tables.

Forward Indexes

Time‑partition pruning (day‑level).

_timestamp column sorted at file and intra‑file levels to enable MinMax‑based data skipping in Trino.

BloomFilter and BloomRangeFilter on frequently filtered columns.

Inverted Indexes

Two custom index types were built on top of Iceberg:

TokenBloomFilter : tokenizes log_msg, builds a Bloom filter per token; low storage, good for low‑frequency terms.

TokenBitMap : leverages Lucene‑style token dictionaries (FST) and per‑token BitMaps of row IDs; provides precise filtering for phrase queries at the cost of higher storage.

Both indexes are generated by Spark jobs that consume Iceberg commit logs, optionally compact small files, reorder data (Z‑order, Hilbert‑curve), and clean up expired snapshots.

Performance Test

Using a 330 GB ORC log dataset:

TokenBloomFilter size: 2.1 GB.

TokenBitMap size: 76.6 GB (high‑frequency tokens omitted).

Sample queries:

select * from test where arg_trid = '1007997177f95bd44536bb570fd193830ab1' and (log_date = '20230512' or log_date = '20230513') order by _timestamp desc limit 200;

Benchmark charts (omitted here) show TokenBitMap outperforms TokenBloomFilter for low‑ and medium‑frequency terms, while for high‑frequency terms the Bloom filter is retained to save storage.

Additional Capabilities

Log Clustering : lightweight clustering algorithm based on fixed‑depth parse trees, line‑number features, and token similarity; provides near‑real‑time grouping of similar logs for faster MTTR.

Computation Push‑down : simple transformations (parsing, filtering) are executed in the log‑agent, reducing backend load.

Scheduling & Partition Commit : high‑priority, dedicated, and general Kafka clusters enable elastic scaling; Flink‑based partition‑commit policies notify the Archer DAG scheduler when Iceberg partitions become ready.

Overall Benefits

The evolution to Billions 3.0 reduced storage cost by at least 20%, improved system stability (no major outages in six months), and delivered sub‑second query latency for hot data while supporting interactive analysis of petabyte‑scale cold logs.

Future Directions

Smooth multi‑cluster ClickHouse sharding.

Insight services for automated log cost‑optimization.

Tighter integration with OpenTelemetry for root‑cause analysis.

Cloud‑native deployment packages for overseas regions.

Dedicated Iceberg meta‑service and index service.

More elastic data pipelines and WASM‑based log‑agent operators.