A Clear Explanation of Data Middle Platforms: Design, Build, Avoid Pitfalls

1. What a Data Middle Platform Is

A data middle platform is a unified data capability platform whose core task is to gather scattered data from various enterprise systems, process it into reusable, standardized capabilities, and continuously support business decisions and innovation. The platform emphasizes continuous data flow rather than static storage.

Key characteristics:

Empowering users : Consolidated global data enables non‑technical roles such as operations, marketing, and supply chain to directly access the data they need.

Capability abstraction : Raw data is transformed into reusable services like customer profiles, sales forecasts, and inventory analysis, allowing one‑time construction for multiple uses.

Shared reuse : Abstracted capabilities can be invoked by any department, eliminating data silos.

2. Six Core Modules of the Architecture

2.1 Data Ingestion and Access

Enterprise data sources are diverse—relational databases, business logs, IoT devices—varying in format and quality. The primary challenge is to reliably and completely ingest heterogeneous data.

Typical technology choices:

Apache Kafka : Real‑time stream processing for high‑concurrency, low‑latency scenarios such as user behavior logs.

Sqoop : Batch synchronization of traditional databases (MySQL, Oracle) for stable offline data transfer.

Flink : Unified stream‑batch engine that handles both real‑time streams and offline batch jobs, offering flexibility for complex analytics.

Selection should match data volume and latency requirements; there is no absolute right or wrong choice.

2.2 Data Storage and Computing

Ingested data is stored in layered fashion, typically a combination of a data lake (low‑cost raw storage) and a data warehouse (structured aggregation for analysis).

Two mainstream compute architectures:

Lambda architecture : Parallel real‑time and batch processing, balancing cost and coverage; recommended for small‑to‑medium enterprises.

Kappa architecture : Pure stream processing for ultra‑low‑latency needs (e.g., real‑time risk control), but with higher operational complexity and cost.

The author advises prioritizing Lambda for most mid‑size companies due to lower risk.

2.3 Data Governance and Security

Governance is the most easily underestimated and error‑prone layer. It comprises four core actions: metadata management, data lineage tracking, quality monitoring, and standardization.

Metadata management reveals the source and meaning of each data item.

Data lineage enables rapid root‑cause analysis when issues arise.

Quality monitoring ensures accuracy and completeness.

Standardization guarantees consistent definitions across sources.

Security basics include RBAC permission control and dynamic masking of sensitive data.

2.4 Data Serviceization

After governance, data is packaged into standardized API interfaces (e.g., user‑profile API, inventory‑forecast API) so business units can consume results without dealing with underlying structures.

2.5 Organization and Team

Successful platforms require deep collaboration between technology and business. Recommended roles:

Data Engineer : Implements technical solutions.

Data Product Manager : Translates business needs into data requirements.

Governance Committee : Oversees cross‑department coordination and rule definition.

2.6 Performance Optimization

As data volume and request concurrency grow, two practical optimizations are suggested:

Cold‑hot data separation : Cache hot data in Redis; store cold historical data in low‑cost object storage such as OSS.

Dynamic compute resource scheduling : Use Kubernetes for elastic scaling, expanding during peaks and shrinking during troughs to avoid waste.

3. From 0 to 1: Four Implementation Stages

3.1 Requirement Analysis & Goal Setting

Start by identifying core business scenarios (e.g., marketing analysis, supply‑chain optimization, customer profiling) and set measurable targets such as 90% data sharing rate or 50% improvement in analysis efficiency.

Assess challenges like data silos and departmental cooperation willingness; pilot 1‑2 high‑value scenarios before full rollout.

3.2 Data Governance

Establish order through:

Defining data standards (field meanings, formats, value ranges).

Building a data dictionary for metadata.

Clarifying data permissions (ownership, access, modification rights).

Data cleaning to remove duplicates, errors, and incompleteness.

Data integration via ETL or APIs to unify sources.

Tools such as FineDataLink can assist in integrating ERP/CRM data, cleaning, and standardizing.

3.3 Data Modeling

Transform governed data into analysis‑ready structures. Key actions include:

ETL processing to convert raw data into analytical models.

Choosing modeling tools (e.g., PowerDesigner, Erwin) for efficiency and quality.

Version‑controlling data models to track changes and rationales.

Documenting model design, business logic, and usage guidelines.

3.4 Data Application

Deliver value to business through:

Understanding specific data needs of each department.

Specifying data type, scope, and quality requirements.

Developing data APIs for query, retrieval, and analysis.

Creating visual dashboards with low‑code tools like FineBI for non‑technical users.

4. Common Pitfalls to Avoid

Business‑technology disconnect : Platforms must be driven by business needs, not built in isolation by tech teams.

Ignoring unstructured data : Text, images, and other unstructured formats often exceed 80% of data and require dedicated storage and analysis solutions.

Over‑ambitious “big‑and‑all” approach : Focus on 1‑2 high‑value scenarios first; expand gradually to reduce risk and shorten cycles.

In conclusion, a data middle platform is a long‑term engineering effort, not a one‑off project. Continuous iteration, quality improvement, and breaking new data barriers are essential for turning data into a reliable decision‑support asset.