Apache Doris Data Lake Federation Features Overview

With the rapid development of data lake technology, query performance has become the main bottleneck for extracting value from massive data. Apache Doris offers a high‑performance query engine that sits on top of a data lake, providing a seamless lake‑warehouse integration that enables fast, easy analytics.

The core ideas behind Doris’s lake‑warehouse integration are fourfold: (1) accelerate lake queries using Doris’s MPP vectorized engine; (2) provide a unified data analysis gateway that abstracts heterogeneous sources; (3) unify data integration by syncing and processing external data within Doris; and (4) create an open data ecosystem that supports open formats like Parquet/ORC and metadata services such as Iceberg, Hudi, Hive Metastore, AWS Glue, and Alibaba Data Lake Formation.

Doris now supports a wide range of data‑lake sources. It can connect to Hive (ExternalTable, ManagedTable) with automatic metadata sync, Iceberg V1/V2 with time‑travel queries, Hudi with copy‑on‑write and merge‑on‑read, and Elasticsearch via a dedicated catalog that auto‑maps indices. JDBC catalog support enables connections to MySQL, PostgreSQL, Oracle, etc., allowing these sources to be treated as dimension tables.

Metadata connection is built on a three‑level hierarchy (catalog → database → table). A unified metadata layer masks differences across sources, while an extensible connector framework simplifies adding new sources. Doris provides efficient metadata caching, real‑time sync via refresh commands or event‑driven updates, and customizable authentication plugins (e.g., Apache Ranger) for fine‑grained access control.

For data access, Doris rewrote the Parquet reader in C++ to eliminate extra memory format conversions, leverage page indexes, Bloom filters, dictionary encoding, and delayed materialization. It also introduced a local file cache with block‑level granularity and consistent hashing to keep cached data close to the query executor, reducing remote I/O and improving stability.

The execution engine was refactored so that the scan layer handles source‑specific logic while the upper planning layer remains identical for internal and external tables, enabling full use of Doris’s optimizer, predicate push‑down, runtime filters, and resource‑aware scheduling. Benchmarks on ClickBench and TPC‑H show 3‑10× performance gains over Trino.

In a financial risk‑control case, Doris unified queries across Elasticsearch, Hive, and Greenplum, eliminating the need for costly data movement and improving timeliness. It also provided a single SQL interface for semi‑structured ES data, streamlining the data‑warehouse service.

Future community plans focus on expanding source support (e.g., Delta Lake, Paimon), enhancing data integration with CDC and materialized views, and introducing pipeline‑based resource isolation and scheduling. Doris 2.0 will also bring compute‑only nodes for elastic scaling and K8s deployment.

Q&A highlights include: Doris cannot yet replace Presto seamlessly but aims to; current BE nodes cannot write back to warehouses, only export data via SELECT ... INTO OUTFILE or EXPORT ; and while no direct Spark comparison exists, Doris is expected to outperform Spark in many OLAP scenarios.