Real‑time Data Lake Architecture with Flink and Hudi: Addressing Timeliness, Observability, and Cost Efficiency

In large‑scale data scenarios, the company operates two parallel pipelines: a high‑timeliness Kafka + Flink stream for real‑time data and a lower‑timeliness Spark batch for offline processing. Existing workflows suffer from latency, weak observability, duplicated resources, and data‑island issues.

To overcome these pain points, a real‑time data lake based on Flink + Hudi is introduced. Hudi provides incremental storage, minute‑level computation, and table‑level query capability, improving both timeliness and observability while unifying real‑time and offline workloads.

DB Ingestion Scenario : MySQL data is periodically synced to the warehouse via DataX. Traditional approaches (DataX → Hive, Canal/CDC → Hudi, Hudi → Hive) each have drawbacks such as full‑day snapshots, lack of repeatable reads, or data redundancy. The proposed Hudi Snapshot View adds filter logic at the meta‑level to isolate specific event‑time partitions, eliminating cross‑day data contamination.

Event‑Tracking (埋点) Scenario : Billions of user‑behavior events are collected daily. By routing these events directly into Hudi tables and exposing them through permission‑aware Views, the solution achieves fine‑grained isolation, minute‑level latency, and reduced I/O via clustering, indexing, and data‑skip techniques.

BI Real‑time Reporting Scenario : Replacing Kafka with Hudi enables unified queries for both real‑time and batch reports, reducing duplicated pipelines and simplifying alerting. However, raw Hudi tables can cause read amplification; to mitigate this, a Flink‑driven Projection materialized view pre‑aggregates data, drastically cutting query latency.

Projection Materialization : Users submit SQL queries that are parsed to create Projection tasks. Incremental Flink jobs compute and store results in materialized tables; the engine rewrites queries to read from these tables when watermarks indicate freshness, otherwise it falls back to the source table, ensuring reliability.

Infrastructure Optimizations : Table Service workloads (Compaction, Clustering) are decoupled from write paths, running on dedicated resources. Hudi Manager automates service orchestration, supports dynamic configuration for OLAP vs. ETL workloads, and contributes enhancements back to the open‑source community.

Finally, the article outlines future directions: intelligent data‑layer optimization, deeper Hudi integration for AI workloads, and continued contributions to Hudi’s core components such as Meta Store, Table Service, and join capabilities.