Applying Data Lake (Hudi) at Kuaishou: Architecture Evolution, Use Cases, and Lessons Learned

The presentation begins with an overview of the data lake (Hudi) adoption at Kuaishou, describing the business challenges of ever‑growing data warehouse size, cross‑department collaboration inefficiencies, and real‑time vs. offline data discrepancies.

To address these issues, Kuaishou evolved its architecture from Hive to Hudi, selecting Hudi for its rich feature set, strong compatibility with the existing big‑data stack, and lower operational cost.

After choosing Hudi, the team designed a wide‑table data model, split tasks according to SLA requirements, and leveraged Hudi's update‑write capability to reduce model complexity, storage, and compute costs while improving data freshness.

Several concrete use cases are presented: (1) CDC data synchronization that reduces latency from 60‑90 minutes; (2) batch‑stream combined processing that enables minute‑level activity calculations; (3) architecture upgrades that consolidate 71 models into three entity‑centric models, yielding cost and performance gains.

The promotion strategy involved validating functional coverage, demonstrating universal applicability across business lines, building a toolchain ecosystem, and encouraging cross‑team collaboration to break silos.

Key reflections include the importance of demand‑driven technology adoption, establishing clear data governance standards, and fostering collective intelligence to ensure successful rollout.

A Q&A section addresses Hudi query optimization, highlighting merge‑on‑read vs. copy‑on‑write modes, incremental consumption, and secondary indexing for faster queries.

The talk concludes that Kuaishou's Hudi journey delivered significant efficiency, cost, and scalability benefits, offering valuable insights for other organizations seeking to modernize their data infrastructure.