Real-Time Data Lake Practices at ByteDance: Architecture, Challenges, and Solutions

ByteDance presents a comprehensive overview of its real‑time data lake, beginning with a historical perspective on the term "data lake" and how its interpretation has shifted from simple centralized storage to sophisticated platforms supporting ACID transactions, streaming, and AI workloads.

The company identifies six essential capabilities for a modern data lake: efficient concurrent updates, intelligent query acceleration, unified batch‑stream storage, unified metadata and permissions, extreme query performance, and AI + BI integration.

Four major challenges encountered during deployment are discussed: difficulty in data management, weak concurrent update support, poor update performance, and cumbersome log ingestion. For each, ByteDance details concrete mitigation strategies.

To address metadata fragmentation, ByteDance built the Hudi Metastore Server, a multi‑tenant service that centralizes metadata, offers Hive‑compatible APIs, and improves query latency. Concurrency issues were solved by adding optimistic locking on top of Hudi’s timeline, enabling both row‑level and column‑level parallel writes, and introducing conflict‑resolution mechanisms inspired by Git.

Performance bottlenecks caused by Bloom‑filter false positives led to the development of a scalable bucket index, which hashes primary keys into file groups, dramatically reducing lookup costs and supporting both batch and streaming workloads. The bucket index was contributed back to the open‑source Hudi project.

Four practical use‑case stages are illustrated: (1) near‑real‑time data validation for e‑commerce pipelines, (2) real‑time data ingestion and interactive analytics with Presto, (3) multi‑dimensional real‑time aggregation for dashboards and data services, and (4) real‑time data joining for feature engineering in recommendation systems.

Future roadmap focuses on three dimensions: functional enhancements (metadata‑level acceleration, data‑level acceleration, index acceleration, and intelligent table optimization), open‑source contributions (enhanced Hudi features, connectors for Trino/Presto, and RPC‑based lake‑table services), and commercial products (LAS lake‑warehouse service and EMR stateless cloud‑native warehouse).

The session concludes with a Q&A covering the scalable bucket index design, real‑time warehouse applicability, schema‑on‑read evolution, and data‑lake layering considerations.