OushuDB: A Cloud‑Native Real‑Time Lakehouse Database – Architecture, Evolution and Practice

OushuDB is a cloud‑native database designed to realize a real‑time, multi‑engine lakehouse system. The article first outlines the historical milestones of cloud‑native real‑time lakehouses, from the early OLTP‑focused database era, through distributed databases, big‑data platforms, lake‑warehouse separation, to the emergence of cloud‑native integrated architectures.

The cloud‑native lakehouse architecture is described as a unified platform that combines data lake storage and data warehouse processing, supporting both real‑time and batch workloads, multi‑modal storage (object storage, HDFS, Magma), and multiple compute engines sharing a single data copy.

Key features of OushuDB include:

Full real‑time data ingestion and integration.

Unified metadata management.

Native support for multiple storage backends (OSS, COS, HDFS, Magma).

Shared compute engines to avoid data silos.

Elastic resource management with on‑demand scaling.

High‑performance, fully real‑time analytics.

The system’s architecture is layered: a client layer (JDBC/ODBC), a stateless master‑node layer for query compilation and routing, a virtual compute cluster layer with stateless compute nodes, and a virtual storage cluster layer that abstracts object storage, HDFS and Magma via a virtual storage sub‑cluster (VSC) mechanism.

Practices for building applications on OushuDB focus on compute‑storage separation, dynamic scheduling close to data, managing the mapping between compute and storage layers, handling node failures, and ensuring read‑write consistency (including ACID transactions via Raft and multi‑replica metadata).

Additional capabilities include native support for the Hudi table format (enabling multi‑engine data sharing), seamless Hive metastore integration for metadata sharing and column‑level security via Ranger, and compatibility with Hive partitioned tables.

Overall, OushuDB demonstrates how cloud‑native technologies, distributed storage, and modern compute frameworks can be combined to build a scalable, high‑availability, real‑time lakehouse solution.