How DragonF MPP DB Redefines Cloud‑Native Data Warehousing at Massive Scale

Speaker and Context

Chen Xiaoxin, product lead for the MPP DB at Jiànxìn Jīnkē Financial Technology Foundation, presented the DragonF MPP DB at the 13th China Database Technology Conference (DTCC2022). The talk covers the system’s architecture, key capabilities, operational results, and future development plans.

R&D Background and Traditional MPP Pain Points

During two decades of data‑warehouse construction, Jiànxìn Jīnkē faced several challenges with conventional MPP databases that use a tightly coupled compute‑storage model:

Complex deployment and management of physical or virtual machines.

Application‑centric “chimney” deployments causing severe data silos.

Redundant data stored across clusters, leading to high resource consumption.

Heavy inter‑cluster data replication consuming network, ETL, and other resources.

To address these issues, the team built DragonF MPP DB with a cloud‑native, compute‑storage‑separated architecture.

Architecture Overview

DragonF MPP DB separates metadata, compute, and storage into three independent layers:

Metadata layer : Managed via an IaaS‑connected console, enabling one‑click cluster creation, start/stop, upgrade, scaling, and self‑healing.

Compute layer : Stateless, horizontally scalable compute clusters that can expand from 100 to 20,000 nodes.

Shared storage layer : Object storage that provides a single source of truth for data, eliminating redundancy and supporting massive concurrent workloads.

Since its first production deployment in March 2020, the system has grown to over 27,000 nodes, handling 18 PB of data, millions of daily jobs, and tens of millions of SQL executions.

Core Technical Features

The logical architecture consists of a management module and a user module. The management module handles resource provisioning, cluster lifecycle, and monitoring. The user module is divided into three layers:

Metadata layer using ETCD for scheduling and service discovery, and FDB for persistent metadata storage.

Compute layer offering completely stateless compute services; each compute cluster operates as an isolated database service that can be created, deleted, or scaled on demand.

Shared storage layer built on object storage, providing high‑throughput, high‑availability, and durable data persistence.

Additional capabilities include built‑in GIS, Python, and other components that enable SQL analytics, machine learning, and spatio‑temporal analysis. The system integrates with external engines such as Hive, Spark, Flink, and Kafka via high‑performance connectors, supporting data federation and federated computing.

Operational Benefits

Cloud‑native design eliminates traditional MPP constraints, allowing independent scaling of compute and storage.

Stateless compute nodes and shared storage enable minute‑level node recovery and self‑healing.

Dynamic scaling and one‑click operations improve operational efficiency by more than tenfold.

Performance gains of over 20% and storage cost reductions of 30% compared with legacy MPP solutions.

Future Outlook

Planned enhancements aim to provide:

Cross‑region high‑availability and disaster‑recovery with multi‑region, multi‑version data storage.

Multi‑tenant metadata services with load control and fault isolation.

Unified handling of structured, semi‑structured, and unstructured data, supporting major open storage formats and data‑lake standards.

Efficient parallel querying by external compute engines and seamless write‑back to the database.

Rich external data source integration and federated computing capabilities.

Broad service offerings including SQL, AI, graph, time‑series, and streaming analytics.