Real-time Precise Deduplication Using StarRocks Materialized Views at Didi

Didi leverages StarRocks materialized views with a global dictionary and bitmap aggregation to perform real‑time, high‑cardinality precise deduplication, automatically rewriting queries and refreshing views, cutting query latency by ~80%, reducing resource use ~95%, and boosting concurrent QPS up to 100‑fold, while planning further automation and bitmap optimizations.

Big DataMaterialized ViewsOLAP

0 likes · 19 min read

StarRocks

Feb 27, 2024 · Databases

How StarRocks Materialized Views Enable High‑Concurrency Precise Deduplication

StarRocks’ materialized view feature lets Didi replace costly fuzzy deduplication with precise, high‑concurrency deduplication for real‑time dashboards, using global dictionary mapping, layered ODS/DWD/ADS views, synchronous and asynchronous refreshes, and transparent query rewrite to cut query latency by 80% and boost QPS dramatically.

Big DataMaterialized ViewsOLAP

0 likes · 20 min read

How StarRocks Materialized Views Enable High‑Concurrency Precise Deduplication

Huolala Tech

Oct 13, 2022 · Big Data

How Druid Uses Bitmap Indexes for Fast Queries and Precise Deduplication

This article explains how Apache Druid builds and queries bitmap indexes for efficient OLAP analysis, and describes a dictionary‑encoding plus bitmap solution—adapted from Kuaishou—to achieve exact deduplication even on high‑cardinality dimensions.

Bitmap IndexDictionary EncodingDruid

0 likes · 14 min read

How Druid Uses Bitmap Indexes for Fast Queries and Precise Deduplication