Building a Data Middle Platform from Scratch: Architecture, Implementation Phases, and Pitfall Avoidance

I. The Essence of a Data Middle Platform

A data middle platform is a unified data capability platform for enterprise digital operations. It aggregates and governs global data according to standards, provides standardized data capabilities and tools to business units, and manages abstraction, sharing, and reuse of data capabilities at the company level. It is not merely a big data platform but a mechanism to "keep data continuously usable." Through standardization and service-oriented delivery, it embeds data capabilities into business scenarios to support rapid decision-making and innovation.

Core Characteristics

Empowering users: Leverages globally aggregated data to enable business use.

Abstraction of data capabilities: Integrates and processes data into reusable capabilities such as customer profiles and sales forecasting.

Sharing and reuse: Toolchains let departments share abstracted capabilities, eliminating data silos and duplicate construction.

Core Functions

Data integration: Collects and unifies data from sources like ERP, CRM, and financial systems into a central repository for consistent management.

Data governance: Cleanses, classifies, and annotates data to ensure quality and usability.

Data services: Exposes standardized APIs so front-end applications can consume data on demand.

Value Delivered

Establishes enterprise data standards (access, modeling, storage, security).

Drives formation of dedicated mid-platform organizations (build team, ops, data product managers, data operations).

Enables cost reduction and efficiency gains through efficient data utilization.

Differentiation from Related Concepts

vs. Data Warehouse: Warehouses serve reporting and decision analysis; the middle platform also serves front-line business services for innovation and rapid market response.

vs. Business Middle Platform: Business mid-platform abstracts shared business processes; data mid-platform abstracts shared data capabilities.

vs. Data Lake: A data lake stores structured and unstructured raw data; the middle platform is a larger system that includes lake storage plus governance, services, and business alignment.

vs. Big Data Platform: Big data platforms focus on storage and compute; the middle platform focuses on data assetization and servitization across governance, security, and the full application chain.

II. Six Core Architectural Modules

1. Data Ingestion & Access: From "Messy" to Highly Available

Supports multi-source heterogeneous data (databases, logs, IoT devices) with both real-time and batch collection.

Technology choices: Apache Kafka for real-time streams, Sqoop for batch sync, Flink for unified stream-batch processing.

2. Storage & Compute: Layered Design Determines Scalability

Classic layered architecture:

Storage layer: Data lake (low-cost raw storage) + Data warehouse (structured aggregates).

Compute layer: Lambda architecture (hybrid real-time + offline) vs. Kappa architecture (pure streaming).

Selection guidance: Small-to-medium enterprises should prefer Lambda to balance cost and scenario coverage.

3. Data Governance & Security: Where 90% of Enterprises Stumble

Governance four pillars: metadata management, data lineage, quality monitoring, standardization.

Security design: RBAC-based access control, dynamic masking of sensitive data (e.g., Alibaba Cloud DataWorks).

4. Data Servitization: From "Warehouse" to "Business Enablement"

API-fication: Encapsulate data as reusable service interfaces (e.g., user profile API, inventory forecasting API).

Toolchain: Qidianyun DataSimba provides end-to-end support from development to API publishing.

5. Organization & Team: Technical + Business "Double Helix"

Team composition: Data engineers (technical delivery) + Data product managers (business translation) + Governance committee (cross-department collaboration).

6. Performance Optimization: Survival Rules for High Concurrency

Practical techniques:

Cold/hot data separation (e.g., OSS archival + Redis caching).

Dynamic compute scheduling (Kubernetes elastic scaling).

III. Three Phases from 0 to 1

Phase 1: Platform Construction

Requirements analysis & goal setting: Deeply engage business units to identify scenarios needing data support (marketing analysis, supply chain optimization, customer profiling). Avoid "building for building's sake." Set quantifiable, strategy-aligned targets (e.g., 50% query efficiency improvement, 90%+ cross-department sharing rate). Identify challenges like data silos and low business cooperation; validate via small pilots before scaling.

Technology selection & architecture design: Choose stack based on existing capabilities, data volume, and future scalability (Hadoop ecosystem for large volumes; Flink/Kafka for high real-time needs). Design flexible, modular layered architecture (ingestion, storage, compute, application) following high cohesion/low coupling. Avoid technical debt by picking mature, stable technologies.

Phase 2: Data Governance

Define data standards: source, meaning, type, value ranges for consistency.

Manage data dictionary: document meaning, format, relationships for understanding.

Data permission management: clarify ownership and usage rights, prevent misuse.

Data cleansing: remove duplicates, errors, incomplete records to improve accuracy.

Data integration: use ETL tools or APIs to break silos and unify management.

Promote sharing: establish mechanisms for inter-department data collaboration.

Concrete example: Enterprises can use FineDataLink to integrate data from ERP, CRM, and other systems, then cleanse and transform it. For instance, ERP may use one date format while CRM uses another; FineDataLink unifies these for downstream analysis. Its governance capabilities help establish standards that keep monthly, quarterly, and annual reports consistent, providing clear, accurate decision support.

Phase 3: Data Modeling

ETL processing: Use tools like FineDataLink to extract, transform, and load data into model-ready form.

Modeling tools: Employ dedicated tools for model development and management to boost efficiency and quality.

Version control: Track model change history for maintenance and upgrades.

Documentation: Record design, development, and usage details for clarity.

Phase 4: Data Applications

Understand business needs: Collaborate with units to identify scenarios (marketing, customer profiling, supply chain).

Define data requirements: Determine types, scope, and quality thresholds.

Develop data APIs: Provide query, retrieval, and analysis interfaces.

Build visualizations: Create charts and dashboards. The author recommends purchasing professional zero-code tools (e.g., FineVis) rather than building in-house to reduce IT effort. IT develops base templates; business users drag-and-drop to build large-screen dashboards.

IV. Pitfall Avoidance Guide: Lessons from Large Enterprises

Business-technology disconnect: Mid-platform construction must be business-driven, not an IT self-indulgence.

Ignoring unstructured data: Text, images, etc. exceed 80% of data volume; design dedicated storage and analysis modules (e.g., Elasticsearch + object storage).

Over-pursuing comprehensiveness: Initially focus on 1-2 high-value scenarios to avoid resource dilution.

Conclusion: The Data Middle Platform Is Not a Destination

The data middle platform acts as the enterprise's data operating system, integrating massive data and providing efficient support to drive digital transformation and innovation. However, construction is not one-off. As business evolves and data grows, continuous iteration is required to improve quality and usability, deliver more accurate and timely support, promote sharing and circulation, break departmental walls, and make data a true driver of decisions and innovation.