Real-time OLAP with Apache Doris: Architecture, Use Cases, and Optimization at Dingdong Maicai

Dingdong Maicai, a fast‑growing food delivery startup, needed a flexible, multi‑dimensional real‑time analytics platform to support diverse business scenarios such as dashboards, ad‑hoc queries, behavior analysis, and tagging. After evaluating options, they selected Apache Doris as the core OLAP engine for its ability to handle both detailed and aggregated queries with high concurrency.

Selection and Comparison – Doris was compared with ClickHouse on criteria including standard SQL support, join capabilities, high‑concurrency handling, real‑time and batch ingestion, and large‑scale detail data queries. Doris outperformed ClickHouse in most areas, leading to its adoption.

System Architecture – The overall data pipeline integrates Elasticsearch for tag storage, HBase for dimension tables, MySQL for business data, Kafka for streaming, Spark for ad‑hoc processing, and Apache Doris as the central OLAP engine. Offline data flows from source databases via DataX into an ODS layer before loading into Doris; real‑time data is processed by Flink from Kafka and written to Doris or HDFS.

Application Practices

Real‑time data analysis: Doris enables sub‑second query latency for high‑frequency forecasting and supply‑chain adjustments, achieving results within 2 seconds.

B‑end query services: Replacing Spark‑JDBC on Hive with Doris reduced query times from minutes to seconds for user‑profile lookups.

Tag system migration: Moving tag data from Elasticsearch to Doris improved join performance, reduced query time from ~20 minutes to seconds, and leveraged distributed joins (Shuffle, Broadcast, Colocate) with Colocate Join delivering the best performance.

Optimization Techniques

1. Colocate Join – Ensures data locality by aligning Distributed Keys and bucket sequences across tables, eliminating exchange nodes and achieving millisecond‑level query speed. Example query:

select count(psp.product_mongo_id) from profile_table psp
left join station_info_table psi on psp.product_mongo_id=psi.product_mongo_id and psp.station_id=psi.station_id
where psp.pdate='2023-03-16' and psp.four_category='特色醋' and psp.brand_name='宝鼎天鱼' and psp.weight_unit='ml' and psp.pmg_name='粮油调味组';

2. Bitmap Aggregation – Replaces count(distinct) with bitmap_count to accelerate large‑scale distinct counts, reducing query time dramatically.

select event_id, date, bitmap_count(uid) as count
from event
where dt>='2022-06-01' and dt<'2022-06-06' and event_id in (...)
group by event_id, str_to_date(dt,'%Y-%m-%d');

3. Prefix and Bloom‑filter Indexes – Prefix indexes speed up queries on sorted key columns, while Bloom filters on high‑cardinality columns (e.g., bloom_filter_columns="area_name,is_booking,user_source,...") prune data early.

4. Materialized Views (Rollup) – Creating rollup tables with a subset of columns (e.g., dt, event_id, dict_id_bitmap) allows queries to scan minimal data, improving aggregation performance.

Broker Load Workflow Automation – The team built a self‑service pipeline for modeling, configuring, and scheduling Broker Load jobs, reducing operational overhead and enabling hourly or daily data ingestion automatically.

Conclusion and Outlook – Apache Doris has become the backbone of Dingdong Maicai's OLAP layer, supporting thousands of tables, high UV/ PV traffic, and millisecond‑level responses. The company plans to expand Doris usage to more scenarios, adopt new community features, and continue contributing to the open‑source ecosystem.