Why Data Lakes Need Data Warehouses: Evolution of Modern Data Platforms

1. Introduction

We live in a big data era where enterprise data volumes explode. Building a robust data platform is critical for handling massive storage and processing challenges, evolving from data warehouses and data lakes to the modern lakehouse architecture.

Understanding the evolution, key issues, and core technologies behind these approaches helps enterprises design better data platforms, which is the purpose of Baidu Intelligent Cloud's data lake series.

2. The Value of Data

"Data is the new oil." — Clive Humby, 2006

Data must be refined to unlock its true value. Raw data resembles crude oil—valuable but unusable without processing. Enterprises must transform raw data through cleaning, integration, and analysis to drive business growth.

3. Components of a Data Platform

A data platform consists of three core parts:

Storage System – stores raw data for long periods, aggregates disparate sources, and provides a single source of truth.

Long‑term retention of historical data.

Support for distributed source ingestion (e.g., MySQL, Oracle, logs, third‑party datasets).

Logical centralization while allowing physical distribution (multi‑cloud).

Compute Engine – extracts useful information from stored data. Different workloads use different engines: TensorFlow/PyTorch for deep learning, Hadoop MapReduce/Spark for batch processing, Apache Doris for BI.

Compute engines have varying data format requirements. Some (e.g., Hadoop, Spark) accept flexible formats like Parquet, while others (e.g., Doris) enforce specific schemas.

Interface – provides user access, most commonly via SQL, with optional programming APIs for specific engines.

4. Data Warehouse vs. Data Lake

4.1 Data Warehouse

Data warehouses emerged early to support business intelligence dashboards by consolidating data from ERP, CRM, and other systems into a single, historical repository.

They rely on OLAP (online analytical processing) and column‑oriented storage to achieve high‑performance, massively parallel queries (MPP). Data is stored using a "Schema‑on‑Write" approach, requiring ETL to transform raw data into a predefined schema.

4.2 Data Lake

Data lakes retain raw data in its original format—structured, semi‑structured, or unstructured—using "Schema‑on‑Read". This approach preserves information for future analysis but can lead to "data swamp" problems if not managed.

Modern data lakes leverage cloud object storage for virtually unlimited, low‑cost capacity, enabling separate scaling of compute and storage (the "compute‑storage separation" architecture).

5. Modern Data Platform: Lakehouse

5.1 Challenges of Data Lakes

Key issues include data quality, metadata management, versioning, and data flow across heterogeneous compute engines. Solutions involve adding ETL layers, robust metadata catalogs, incremental update mechanisms (e.g., Apache Iceberg, Hudi, Delta Lake), and standardized data formats.

5.2 Convergence of Lake and Warehouse

Practices from data warehouses—ETL, ACID guarantees, access control—are being applied to data lakes, blurring the boundaries. SQL remains a preferred interface, and data warehouses increasingly support lake data formats.

5.3 Lakehouse Architecture

The lakehouse combines object‑storage‑based data lakes with metadata and acceleration layers, supporting data warehouse, big data, AI, and HPC compute engines via multiple interfaces (SQL and others).

Accelerated storage (high‑performance file systems or caches) addresses latency‑sensitive workloads, while unified metadata ensures data discoverability, lineage, and fine‑grained access control.

6. Summary

Enterprise data growth drives continuous innovation in data platforms. Data warehouses and data lakes each have strengths; their convergence into lakehouse architectures offers a one‑stop solution that leverages open‑source technologies, cloud‑native object storage, and flexible compute engines to meet diverse business needs.