Data Fabric vs Data Mesh: Choosing the Right Architecture for Modern Big Data Platforms

Background

Big data platform construction is intrinsically complex and constantly evolving, moving from traditional warehouses to Data Lakes and LakeHouses, with a myriad of batch, streaming, MPP, and machine‑learning engines. Organizations face technical, organizational, and methodological challenges such as component selection, architecture design, performance analysis, ongoing operations, scaling, and stability, often resulting in multiple co‑existing platforms and fragmented data.

Focus on Data Fabric and Data Mesh

The article concentrates on clarifying the often‑confused concepts of Data Fabric and Data Mesh, explaining the problems they aim to solve, their architectural characteristics, viable technology stacks, maturity gaps, and their relationship to our big‑data services.

Big Data Technology Stack

System platforms: Hadoop, CDH, HDP

Cloud platforms: AWS, GCP, Microsoft Azure

Monitoring: CM, Hue, Ambari, Dr.Elephant, Ganglia, Zabbix, Eagle, Prometheus

File systems: HDFS, GPFS, Ceph, GlusterFS, Swift, BeeGFS, Alluxio, JindoFS

Resource schedulers: K8s, YARN, Mesos, Standalone

Coordination: ZooKeeper, Etcd, Consul

Data stores: HBase, Cassandra, ScyllaDB, MongoDB, Accumulo, Redis, Ignite, Geode, CouchDB, Kudu

Columnar formats: Parquet, ORC, Arrow, CarbonData, Avro

Data lakes: Iceberg, Hudi, DeltaLake

Processing engines: MaxCompute, Hive, MapReduce, Spark, Flink, Storm, Tez, Samza, Apex, Beam, Heron

OLAP: Hologres, StarRocks, Greenplum, Trino/Presto, Kylin, Impala, Druid, Elasticsearch, HAWQ, Lucene, Solr, Phoenix

Ingestion: Flume, Filebeat, Logstash, Chukwa

Data exchange: Sqoop, Kettle, DataX, NiFi

Messaging: Pulsar, Kafka, RocketMQ, ActiveMQ, RabbitMQ

Scheduling: Azkaban, Oozie, Airflow, Crontab, DolphinScheduler

Security: Ranger, Sentry, Atlas

Lineage: OpenLineage, Egeria, Marquez, DataHub

Machine learning: PAI, Mahout, MADlib, Spark ML, TensorFlow, Keras, MXNet

Typical open‑source stack combinations include Iceberg+S3+StarRocks+Flink, HDFS+Alluxio+Spark+Trino, HDFS+Hive+Greenplum, and MinIO+LakeFS+Marquez+Trino.

Concept Analysis

Data Fabric

Conceptually, a Data Fabric provides a metadata‑driven virtual layer that unifies disparate data tools, delivering capabilities such as data access, discovery, transformation, integration, security, governance, lineage, and orchestration.

Positioning: Creates a unified virtual layer that abstracts storage, compute, and MPP databases, allowing read/write and computation to be orchestrated centrally.

Technical elements: Data integration, service integration, unified semantics, active metadata, knowledge graph, intelligent catalog.

Does not require organizational change; data teams can continue to manage platforms.

Data Mesh

Data Mesh emphasizes domain‑oriented ownership, treating data as a product and enabling self‑serve platforms. It encourages distributed teams to manage their own data while adhering to shared governance.

Four main characteristics: domain‑centric ownership, data as product, self‑serve platform, cross‑domain federated computation.

Governance levels range from no analytics (Level 0) to publishing data as a product (Level 4).

Comparison

Both aim to eliminate data silos and provide a self‑serve platform without heavy ETL.

Data Fabric is technology‑centric, building a unified virtual layer; Data Mesh is method‑centric, focusing on organizational change and domain autonomy.

Technical Implementation of Data Fabric

Catalogue

A unified catalogue must abstract the three‑level hierarchy (catalog‑database‑table) across engines. Iceberg, for example, offers multi‑catalog compatibility, but each engine still requires a specific implementation (e.g., iceberg-spark-runtime-3.3_2.12:1.1.0.jar).

@Override
public Database getDB(String dbName) throws InterruptedException, TException {
    org.apache.hadoop.hive.metastore.api.Database db = clients.run(client -> client.getDatabase(dbName));
    if (db == null || db.getName() == null) {
        throw new TException("Hive db " + dbName + " doesn't exist");
    }
    return convertToSRDatabase(dbName);
}

Data Format

Unified columnar formats such as Apache Arrow enable efficient data exchange between engines, reducing serialization overhead.

Lineage & Discovery

Cross‑engine lineage requires a third‑party service (e.g., OpenLineage, DataHub) to aggregate metadata from various engines, enabling full‑pipeline visibility and impact analysis.

Unified Development & Semantics

Tools like dbt provide modular SQL development, but execution still occurs within the underlying warehouse. Trino offers a federated SQL engine that can query across multiple sources, illustrating the distinction between data‑fabric‑style virtual layers and mesh‑style domain autonomy.

Impact on Big‑Data Services

Our services leverage Data Fabric principles to build adapters that bridge heterogeneous platforms, and Data Mesh methodology to design domain‑oriented data products. Typical engagements include:

Migration of heterogeneous data platforms to cloud‑native solutions (e.g., Alibaba MaxCompute).

Planning lake‑warehouse and streaming architectures for co‑existence and gradual evolution.

Optimizing data production and operations through unified lineage, quota analysis, and cross‑domain analytics.

Conclusion

Data Fabric and Data Mesh address data fragmentation from complementary angles: Fabric provides a unified technical virtual layer, while Mesh offers a domain‑centric organizational model. In practice, a hybrid approach—building a virtual layer with Fabric and empowering domains with Mesh—can deliver flexible, scalable, and future‑proof big‑data solutions.