How Distributed Lakehouse Architecture Solves Data Swamp Challenges

1 Related Research

1.1 Definition of Lakehouse

Databricks introduced the lakehouse concept in 2020, describing it as a new open data management architecture that combines the flexibility, cost‑effectiveness, and scale of data lakes with the data management and transaction capabilities of data warehouses. Armbrust et al. define it as a low‑cost, directly accessible storage system that also provides traditional analytical database features such as transaction management, data versioning, auditing, indexing, caching, and query optimization.

1.2 Differences Among Data Warehouse, Data Lake, and Lakehouse

Key differences include data types (structured relational data in warehouses, raw formats in lakes, both in lakehouses), ingestion processes (pre‑defined ETL for warehouses, schema‑on‑read for lakes, hybrid for lakehouses), access methods (SQL‑centric for warehouses, open APIs for lakes and lakehouses), reliability and security (mature and secure warehouses, data‑swamp‑prone lakes, lakehouses add warehouse‑level management and security), and applicable scenarios (BI for warehouses, data science for lakes, both for lakehouses).

2 Distributed Lakehouse Architecture

The distributed lakehouse partitions massive data resources by business domain, allowing domain teams to manage their own data, reducing centralized governance difficulty and improving efficiency. Each domain can deploy lakehouse modules independently, share data through a mesh catalog, and cooperate on joint governance policies, ensuring consistent results across the enterprise.

2.1 Data Domain Decoupling

Data domains are classified into three types: source domains (raw data from business systems or social media), aggregation domains (consolidated data such as user profiles derived from multiple sources), and consumer/user domains (processed data tailored for specific analytical use cases).

2.2 Cross‑Domain Data Sharing

Domain nodes publish metadata to a mesh catalog, which is reviewed and recorded by administrators. When a node needs data, it queries the catalog, sends a request through a data‑sharing service, and the provider routes the data to the requester, continuously monitoring usage to ensure controlled sharing.

2.3 Joint Data Governance

Joint governance combines global governance (defining ownership, common policies, shared vocabularies, minimum quality and security standards) with domain‑specific governance (tailored metadata, quality, and lifecycle management), enabling scalable governance in large‑scale data environments.

3 Lakehouse Functional Modules

The lakehouse consists of three zones: data sources, the core lakehouse functional area, and users. The functional area is divided into six layers: data ingestion, data lake, compute, data service, data analysis, and data governance.

3.1 Data Sources

All systems that generate data can be sources, including OLTP systems (structured relational data), NoSQL stores (key‑value, graph, JSON), and unstructured sources such as text, streaming telemetry, and media files.

3.2 Core Functional Area

Data Ingestion Layer

Supports batch and real‑time ingestion. Batch ingestion periodically extracts data to the lake; real‑time ingestion streams data via services like Kafka, capturing change data capture events.

Data Lake Layer

Organizes data into raw, intermediate, processed, and archive zones. Raw data is stored in its original format (CSV, Parquet, JSON). Intermediate data holds results of cleaning, filtering, or aggregation. Processed data serves BI, reporting, and machine‑learning workloads. Archive zone stores cold data cost‑effectively.

Metadata and API layers sit atop the lake, providing transactional capabilities (via Hive, Delta Lake, Iceberg) and rich SQL and DataFrame APIs for analytics.

Compute Layer

Decouples storage and compute, allowing independent scaling. Includes engines such as Hive, Spark, Flink, and Impala for batch, streaming, and machine‑learning workloads.

Data Service Layer

Delivers processed data through SQL‑based warehouse services, NoSQL APIs, real‑time services, and data‑sharing mechanisms. Supports both SMP (e.g., MySQL) and MPP (e.g., Redshift) architectures depending on scale.

Data Analysis Layer

Provides descriptive analysis (ad‑hoc queries, BI reports, self‑service BI, lake exploration) and advanced analysis (data science, machine learning) via sandbox clusters, BI services, and ML systems.

Data Governance Layer

Ensures reliability, accessibility, and quality through governance strategy, lifecycle management, metadata management, master data management, quality management, and security management (IAM, encryption, data masking).

3.3 Users

Technical users (data scientists, analysts) and functional users (business managers) both benefit from unified access to raw and processed data.

4 Unified Stream‑Batch Data Flow

To meet lakehouse internal data movement needs, an ELTL (Extract‑Load‑Transform‑Load) process is used. Batch data is ingested, distributed across compute nodes, transformed, and stored in the processed zone. Real‑time data is captured via event‑driven middleware, micro‑batched, processed, and immediately pushed to the service layer.

5 Conclusion

Transitioning from a data lake to a lakehouse marks a shift toward structured, systematic data platforms. A distributed lakehouse decouples business data, enabling scalable architectures and advancing data‑governance thinking for massive, heterogeneous environments. Future work will focus on practical implementation methods for the proposed architecture.