Bilibili’s Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practices

Introduction – The speaker, Zhang Chenyi, a senior development engineer at Bilibili, presents the exploration and practice of Bilibili’s Iceberg‑based streaming‑batch integration.

1. Massive User Behavior Data Transmission

Background – User behavior data (logs, behavior, business DB) is collected via log‑agent and bfe‑agent, routed to Kafka, and downstream can be written to Kafka, Hive, or Iceberg.

Iteration – The offline ODS‑to‑DWD path was migrated from Hive to Iceberg, and the processing changed from Spark batch to Flink streaming, leveraging Iceberg snapshots for minute‑level freshness. Flink SQL can be auto‑converted to batch SQL for reuse, and Iceberg views reduce data duplication.

Data Optimization – Magnus, an internal Iceberg optimization service, performs compaction, small‑file merging, sorting, partitioning, indexing, and pre‑computed files to improve batch query performance.

Benefits – The redesign yields ~3.55 M CNY annual cost savings (20% CPU, 22% memory reduction) and a 48.9% average performance improvement.

2. Commercial and AI Online Training

Background – Real‑time and offline training pipelines use Flink double‑stream joins to generate base data, compute features, and feed training samples to Kafka. Issues include divergent storage (Kafka vs Hive), duplicated APIs, and high CPU cost for dumping Kafka data to Hive.

Iteration – Feature output was moved from Kafka to Iceberg, unifying storage and API. Iceberg’s hot‑cold separation stores recent data on SSD. CPU consumption for dump tasks reduced by ~20%.

Planning – Future upgrades to Flink 1.20 aim to replace Kafka base streams with Iceberg, improve checkpoint latency, and handle large keyed state without failures.

3. DB Data Synchronization

Background – Flink CDC streams full and binlog data to Kafka, then writes to Iceberg Append tables. Data is later merged into Hive for user consumption. Traditional dimension‑table joins use Hive with hour‑level latency.

Iteration – Real‑time path writes CDC data directly to Iceberg MOR tables, enabling minute‑level query latency and providing changelog incremental reads. Write operators now include keyed state to avoid replay on restarts.

Stream Read – Describes the three Iceberg file types (Data File, Position Delete File, Equality Delete File) and enhancements to Equality Delete Files to store full row data for CDC use, simplifying changelog reads.

Outlook – Proposes a single‑chain solution offering instant query via Iceberg as a MySQL mirror, incremental changelog reads, and full‑snapshot tags for offline use, with required optimizations (checkpoint‑driven tagging, cache operator, separate tag files).

4. Iceberg Dimension‑Table Join

Background – Existing double‑stream joins to remote DBs cause high QPS pressure, complex connector configuration, and state loss on restarts.

Iteration – Leverages Iceberg changelog incremental reads to build a new lookup join that stores the full dimension table in Flink state (no TTL), supports incremental updates, and preserves primary keys.

Transformation – Introduces SQLHint for join syntax, modifies Calcite/Flink rules to retain primary‑key fields, removes DropUpdateBefore operator, and blocks left‑stream until right‑stream dimension data is fully loaded.

Benefits – After transformation, the dimension‑table join eliminates remote DB pressure, reuses Flink streaming logic for batch back‑fill, and reduces latency dramatically.

5. Q&A

Q: How are conflicts resolved between Magnus and Flink writes?

A: Magnus compaction is now triggered by Flink JobManager on a configurable schedule (e.g., hourly), blocking the main stream only during compaction, which completes within minutes, minimizing impact on DB sync latency.

Overall, the presentation details the end‑to‑end migration from traditional batch‑oriented pipelines to a unified streaming‑batch architecture powered by Iceberg, highlighting concrete performance, cost, and operational benefits.