Kuaishou’s Data Lake Technical Maturity Curve: Challenges and Solutions with Apache Hudi

This article originates from the Data Lake Technical Maturity Curve roundtable.

Q: What problems did Kuaishou aim to solve by introducing a data lake and what is the current status?

A: Jin Guowei, Kuaishou Data BP lead, explained the business‑driven challenges.

First problem: The rapid expansion of the offline data warehouse caused huge storage and compute cost pressure, with a surge in new tables and models that increased maintenance complexity and reduced development efficiency.

Second problem: Multiple similar data models were built for different time points and dimensions, leading to warehouse bloat, redundant storage, and higher maintenance overhead.

Third problem: Ensuring data consistency became difficult as many models needed simultaneous updates when business definitions changed, creating complex dependencies and risk of inconsistencies.

Additional issues included fragmented departmental data requiring extensive coordination, and a gap between real‑time and offline data that affected business decisions.

To address these challenges, Kuaishou evaluated several technologies and selected Apache Hudi as a complementary solution. By leveraging the data lake’s primary‑key design and Hudi’s schema‑evolution feature, they reduced the need for separate models for each time point, significantly easing model construction and management.

On the collaboration side, once the overall schema is defined, each business team can update its own data, enabling rapid troubleshooting when metrics deviate.

To close the real‑time‑offline data gap, they adopted a real‑time lake ingestion approach, writing data into Hudi tables and using the company’s One Service engine for ad‑hoc queries, while also exploring Paimon’s batch‑stream integration to further lower latency.

The above outlines Kuaishou’s practical experience with data‑lake technology.

The Data Lake Technical Maturity Curve includes nearly one hundred technical points covering architecture, design principles, storage, file formats, table formats, core functions, and frontier technologies, each evaluated for maturity, business value, and difficulty; readers are invited to scan the QR code to download the curve, explanatory video, and documentation.