How to Choose and Architect a Data Lake Platform for Enterprise Digital Transformation

Background and Motivation

In recent years, China’s macro‑economy has entered a “new normal” and the state has promoted “Digital China” and the “digital economy” as national strategies. The 2022 “14th Five‑Year” Digital Economy Development Plan emphasizes the role of data as a production factor and calls for accelerating industrial digital transformation. To realize a digital‑group vision, the enterprise must establish a unified data‑governance system, build a data lake, and enable end‑to‑end data collection, governance, sharing and empowerment.

1. Data Lake Platform Selection Requirements

1.1 Storage Requirements

Linear scalability and support for massive structured, semi‑structured and unstructured data.

Lake‑warehouse integration (third‑generation big‑data architecture).

Real‑time ingestion (incremental sync from relational databases, update/delete support).

Batch ingestion from diverse sources (business DB, FTP, message logs, IoT, etc.).

Unified stream‑batch processing with minute‑level end‑to‑end latency.

Distributed storage (HDFS or similar) with REST interface, node grouping for performance.

Data sharding and replica mechanisms for fault tolerance.

Separation of compute and storage, allowing low‑cost object storage.

1.2 Compute Requirements

Multi‑model heterogeneous engines: distributed analytics, stream processing, high‑performance databases, full‑text search.

Support for NoSQL, HiveQL, standard SQL dialects.

Offline batch jobs (Spark+Hive, Flink+Hive) with unified resource scheduling and dynamic scaling.

Real‑time stream processing with exactly‑once semantics, rich time semantics and state management.

High‑performance analytical database for real‑time data‑warehouse use (sub‑second query, high concurrency, complex analysis).

In‑memory distributed execution engine for fast computation and interactive queries, with data locality and unified scheduling across storage engines.

Multi‑tenant capability for different business units.

1.3 Tooling Requirements

Data development, governance and asset‑operation capabilities.

Support for data integration, real‑time and batch synchronization from relational DBs, file systems, APIs, message queues.

Visual development interface, one‑stop orchestration, scheduling and operation‑monitoring.

Data asset catalog with lineage, metadata, security tags and fine‑grained RBAC.

Data quality management, standard management and security management (sensitive data identification, encryption, access control).

2. Data Lake Architecture Design

2.1 Overall Data Architecture

The enterprise adopts a multi‑modal heterogeneous data lake that stores raw, cleaned and modelled data in a unified “lake‑warehouse” to serve high‑concurrency historical queries, real‑time queries, reporting, batch processing and data‑mining.

Stream‑batch convergence is achieved through a unified engine (Flink/Spark) and Table API/SQL, enabling a single job to process both real‑time streams and historical batches with exactly‑once guarantees. The architecture already supports real‑time risk control, recommendation and supply‑chain monitoring, and can be extended to multimodal data processing.

2.2 Data Ingestion and Aggregation

Real‑time capture of financial flows, orders, HR data, OCR‑based invoice recognition, and anomaly alerts.

Offline batch extraction from ERP, budgeting, financial sharing, master data, contracts, etc., followed by cleansing and integration.

2.3 Data Governance and Control

Data quality management to ensure accuracy, completeness, consistency and timeliness.

Data standard management to unify definitions, formats and usage norms.

Data security management with RBAC (e.g., finance data limited to audit role) and privacy protection.

3. Overall Platform Functional Architecture

The platform must satisfy functional and non‑functional requirements: linear storage scalability, powerful compute (offline, streaming, search, distributed execution, multi‑tenant, high‑performance DB), PB‑level real‑time analysis, deployment on domestic servers and operating systems, elastic cluster scaling, support for heterogeneous data sources (relational, NoSQL, domestic databases, HDFS, Hive, search engines), and multi‑instance deployment on a single node.

4. Conclusion

By integrating advanced data‑lake technologies with the group’s specific business characteristics, the enterprise built a highly scalable, highly available and secure data lake platform. The platform unified data ingestion standards, linked over ten subsidiary units, and established a comprehensive data‑asset catalog, quality and metadata management mechanisms. Future work will focus on enhancing data‑asset registration, quality assessment, lifecycle management, and exploring AI‑driven data‑lake capabilities.