How Cloud‑Native Distributed Databases Are Shaping the Future of Enterprise Data

Cloud‑Native Distributed Database Systems Development

The Alibaba CIO Academy organized a public technical training during the national pandemic response, featuring Li Feifei (aka "Feidao"), chief scientist of the DAMO Academy database team, who presented an overview of enterprise‑grade cloud‑native distributed database systems.

Database Market Analysis and Forecast

According to Gartner, the 2018 global infrastructure software market was about $200 billion, with databases accounting for 20% (~$46.1 billion). China's database market is roughly ¥16.1 billion, though likely under‑estimated. In 2018, the database market grew 18%, with cloud databases representing 22.75% and projected to reach 75% within 2‑3 years. Amazon leads the global cloud database market, with AWS Aurora generating $300 million revenue in 2018.

Evolution of Database Systems

From the 1980s commercial start‑up (Oracle, IBM DB2, Sybase, SQL Server, Informix) to the 1990s rise of open‑source databases (PostgreSQL, MySQL) and analytical databases (Teradata, Greenplum). The 2000s saw Google’s NoSQL breakthroughs (GFS, Bigtable, MapReduce) leading to modern NoSQL solutions (MongoDB, Redis). Post‑2010, cloud‑native databases such as AWS Aurora, Azure SQL Database, Google Spanner, Alibaba POLARDB and AnalyticDB emerged, offering integrated distributed, multi‑model, HTAP capabilities.

Databases as a Critical Cloud Component

Databases are the key link in cloud adoption, transitioning from IaaS to intelligent application layers, ensuring data consistency, high availability, and performance for transaction‑heavy workloads.

Database Types from a Business Perspective

Traditional OLTP relational databases – handle high‑concurrency transactions (e.g., banking, e‑commerce).

NoSQL and specialized databases – store semi‑structured or unstructured data (documents, graphs, time‑series) with horizontal scalability.

Analytical OLAP databases – support massive data analysis, deep queries, and integration with transactional systems.

Database management and control tools – data transfer, backup, and governance.

Database control platforms – unified instance management across private, public, hybrid clouds.

Core Modules of a DBMS

A DBMS consists of application interfaces, SQL interfaces, query execution engine (planner, optimizer, executor), data access layer (transaction processing, memory management, security, file/index management), and storage engine (data, index, metadata files). Transaction processing, including crash recovery and concurrency control, is the most critical module.

Query Processing Flow

SQL or Dataframe API statements are parsed by the Simba parser, producing a catalog and logical plan, which is optimized into a physical plan using statistics (indexes, memory). The optimized physical plan is then executed to produce results or RDDs.

Challenges in Database Systems

Key challenges include write‑write conflicts, data consistency during parallel access, and recovery from failures during bulk writes.

Transaction Guarantees

Transactions provide atomicity, consistency, isolation, and durability (ACID), ensuring either full completion or no effect in case of errors.

Relation to Big Data

Analytical databases face challenges such as complex queries for user behavior prediction, requiring integration with machine‑learning models.

Cloud‑Native Architecture Trends

Traditional single‑node architectures lack elasticity. Cloud‑native databases use distributed shared storage (RDMA) and separate compute from storage, enabling elastic scaling, high availability, and multi‑instance management (DBaaS).

Alibaba Cloud Database Portfolio

From 2005‑2009 Alibaba operated the largest Oracle RAC cluster in APAC. Between 2010‑2015 it adopted open‑source databases and sharding. Since 2016 it has developed its own databases: TP‑oriented POLARDB and OceanBase, and AP‑oriented AnalyticDB.

Multi‑Model Databases

Modern databases support multiple storage models (row, column, KV, document, graph) and query interfaces, aiming to handle heterogeneous data within a single engine.

Intelligent & Automated Management

Machine‑learning techniques are applied to monitor performance, detect slow SQL, and enable self‑driving database platforms that automatically adjust resources and optimize workloads.

Hardware‑Software Co‑Design

Future enterprise databases will integrate hardware and software tightly to maximize performance and efficiency.

Row vs. Column Storage

Row storage excels at transactional updates; column storage excels at analytical reads, reducing unnecessary I/O.

HTAP (Hybrid Transactional/Analytical Processing)

HTAP combines row and column storage to support both OLTP and OLAP in a single system, facing consistency challenges.

CAP Theory and High Availability

Distributed databases balance consistency, availability, and partition tolerance, using protocols like Paxos and Raft to achieve consensus across replicas.

Oracle Migration with ADAM

ADAM evaluates compatibility, automates migration planning, and uses DTS for data transfer, providing a standardized Oracle‑to‑cloud migration pathway.

Security and Encryption

Alibaba Cloud employs transparent data encryption (TDE), data‑at‑rest encryption, and optional blockchain‑based audit logs to ensure confidentiality and integrity.

Summary

Enterprise‑grade cloud‑native databases are converging on distributed, elastic architectures with HTAP capabilities, multi‑model support, AI‑driven automation, and robust security, forming a complete ecosystem that underpins Alibaba’s dominant position in the Asia‑Pacific cloud database market.