Data Middle Platform: General Architecture and Core Components

Most enterprises now prefer a centralized data collection and storage model rather than siloed pipelines, enabling rapid application deployment and unified data management as a strategic asset.

The emergence of the data middle platform addresses the mismatch between data development speed and application development, originating from Alibaba's "big middle platform, small front end" concept.

A generic data middle platform architecture decouples six functional subsystems—data storage, acquisition, processing, governance, security, and operation—allowing flexible composition and incremental implementation.

Key components include:

Data storage framework: supports object, block, and database storage, handling raw, structured, and unstructured data, as well as metadata and tagging.

Data acquisition framework: provides multiple ingestion methods (FTP, DB, API, streaming, web crawling) and preprocessing to filter irrelevant data.

Data processing framework: handles ETL, batch and stream processing, AI analysis, cleansing, exchange, and scheduling.

Data governance framework: manages catalogs, models, metadata, master data, quality, and lineage, but excludes security and sharing functions.

Data security framework: implements logging, authentication, authorization, and encryption across all data flows.

Data operation framework: offers portals, capability exposure, data publishing, and monitoring to support external services.

The platform’s ultimate goal is to transform collected data into reusable services, supporting various business scenarios while ensuring security, scalability, and continuous operation.