Databend: Cloud‑Native Modern Data Warehouse Architecture and Features

Databend, whose name derives from the relativistic concept of "Time Bend," aims to let users rethink data to extract greater value.

The article outlines three main topics: (1) why traditional data warehouse architectures no longer meet current needs, (2) the advantages of modern data warehouse designs, and (3) Databend's cloud‑native architecture.

Traditional data warehouses integrate compute and storage on the same machines, leading to costly scaling, low elasticity, and performance degradation as data grows.

Modern data warehouses require five key capabilities: no hardware management, no software configuration, no resource bottlenecks, second‑level elastic scaling, and pay‑for‑use only when resources are active.

Databend satisfies these needs with a cloud‑native design that separates storage, compute, and metadata services, uses shared cloud storage (e.g., S3, Azure Blob), and provides elastic, multi‑tenant meta services.

The architecture consists of four layers: data ingestion (SQL‑compatible), Meta Service (metadata and multi‑tenant key‑value store), compute layer (independent SQL parsing, planning, and execution nodes), and storage layer (cloud object storage with columnar Parquet files, MinMax and sparse indexes).

Query planning in Databend generates logical plans, optimizes them, and produces physical pipeline plans represented as directed acyclic graphs. Example execution plan code:

explain pipeline SELECT avg(age) FROM class WHERE age > 13 GROUP BY city

The physical pipeline creates parallel execution nodes, enabling vectorized processing and work‑stealing task scheduling to keep CPUs busy.

Databend also employs Arrow Flight RPC for efficient data transfer, avoiding serialization overhead, and supports automatic data clustering to accelerate hot‑data queries.

Future plans include early‑stage Alpha releases, continued performance and elasticity improvements, and a move toward serverless operation where compute resources are provisioned on demand and billed per use.