Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

The presentation begins with an introduction to real‑time data warehouses, explaining their purpose and how they differ from traditional offline warehouses, followed by a five‑part agenda covering legacy architectures, issues, new designs, and a Q&A session.

It then describes the previous Lambda‑style real‑time warehouse used at JD Technology, outlining its data link, computation layer, and storage layer, and highlighting problems such as tight coupling of services, limited visibility, and user expectation mismatches.

The next section examines the drawbacks of the older architecture, including advantages of independent development and disadvantages like inherited Lambda complexities and ambiguous real‑time warehouse positioning.

Subsequently, the new data‑lake‑based solution is introduced, detailing how the data layer now leverages binlog and lake tables, the separation of real‑time and product layers, and the iterative rollout across three major versions, each addressing specific technical and non‑technical challenges.

Further discussion covers the benefits achieved by adopting the lake‑warehouse integration, such as reduced complexity, faster delivery cycles, clearer boundaries between offline, real‑time, and near‑real‑time processing, and improved alignment with business needs.

The article concludes with a summary of the new architecture’s issues, future expectations, and a Q&A segment addressing topics like CK vs. OSS, Paimon adoption, and component choices between Spark and Flink.