This article walks through why using a Redis Set for 200 million daily active users consumes 6‑8 GB of memory, how switching to a sharded Bitmap reduces usage to about 24 MB, and how Roaring Bitmap and ClickHouse further optimize sparse data and analytical workloads.
Facing minute‑level decision demands and billions of marketing events during Taobao's Flash Sale, the Ele.me data team built a real‑time lakehouse with StarRocks and Paimon, leveraging asynchronous materialized views, RoaringBitmap de‑duplication, and resource isolation to achieve sub‑second query latency, lower storage costs, and stable high‑concurrency.
Faced with billions of marketing events and minute‑level decision requirements during Taobao's flash‑sale campaign, the e‑commerce data team built a real‑time lakehouse using StarRocks and Paimon, leveraged asynchronous materialized views and RoaringBitmap deduplication, and achieved sub‑second query latency, massive cost savings, and stable high‑concurrency performance.
This article explains how a CDP system tackles the storage and set‑operation challenges of billions of user‑ID tags and groups by adopting bitmap techniques, especially RoaringBitmap, to dramatically reduce space usage and enable fast union, intersection, and difference calculations.
This article explains how a CDP tackles billions‑scale user ID tags and groups by replacing naïve text‑file storage with bitmap techniques, detailing Bitmap basics, encoding strategies, Java BitSet limitations, and the adoption of RoaringBitmap for efficient compression and fast set operations.
This article explains how Lazada’s internal selection platform leveraged Hologres and the RoaringBitmap compression algorithm to dramatically reduce storage costs, accelerate set operations, and break the 200,000‑item pool limit, achieving up to a 99% speed improvement in scheduling.
This article examines how bitmap data structures and multithreading can dramatically accelerate blacklist filtering in large‑scale marketing systems, reducing processing time from tens of minutes to milliseconds while saving memory and improving overall system performance.
This article explores how to achieve fast, scalable data retrieval in million‑level high‑concurrency scenarios by replacing naïve full‑combination rule matching with column‑wise inverted indexes and bitmap operations, dramatically reducing time complexity and improving stability while leveraging RoaringBitmap compression for space efficiency.
This article explains how Alibaba's user growth platform leveraged RoaringBitmap in Hologres to accelerate massive user profiling, reducing analysis time from several minutes to around ten seconds by redesigning bitmap storage, optimizing data pipelines, and employing efficient SQL and scheduling techniques.
The article explains how Hologres RoaringBitmap compresses user ID sets into efficient bitmap indexes, splits 64‑bit IDs into buckets, syncs them from MaxCompute, and enables sub‑second user portrait queries that previously took minutes, dramatically improving performance and scalability.
This article details Meituan's search‑engine team optimization of Elasticsearch for a high‑traffic LBS scenario, describing the performance bottlenecks in term‑posting retrieval and merging, the design of a run‑length‑encoding (RLE) inverted index, its integration as a plugin, extensive benchmarking, and the resulting 84% reduction in TP99 query latency.
Meituan's food‑delivery search team identified heavy CPU and latency hotspots in Elasticsearch's posting‑list query and merge phases, then redesigned the inverted index using Run‑Length Encoding, hash‑based term lookup, index sorting and a custom SparseRoaringDocIdSet, ultimately reducing TP99 search latency by 84% and cutting CPU usage dramatically.
RoaringBitmap improves traditional BitMap by lazily allocating four container types, compressing sparse data, and dynamically switching between array, bitmap, and run containers, enabling fast exact set operations that power big‑data systems such as Kylin, ClickHouse, and B‑Station’s user‑visit and crowd‑package pipelines, dramatically reducing memory use and processing latency.
This article explains the principles, Java implementation, and performance benefits of Bitmap and RoaringBitmap, demonstrating how they dramatically reduce storage costs, enable fast deduplication and set operations, and optimize large‑scale data warehouse queries in real‑world scenarios.
By representing pre‑and post‑refresh SPU sets as RoaringBitmaps and diffing them, the system avoids full‑insert writes, cuts memory usage by orders of magnitude, speeds refreshes by over 50 % and reduces write volume nearly 87 %, solving large‑scale tag‑based product refresh challenges.
For deduplicating game recommendations in Redis, RoaringBitMap outperforms intset and Bloom filters by storing 300 auto‑incrementing game IDs in roughly 0.5 KB—over twice the compression of intset and far smaller than the 29 KB Bloom filter—thereby cutting memory use, latency, and hardware costs.
This article explains how to implement both offline (T+1) and real‑time UV counting using Hologres with RoaringBitmap for high‑cardinality aggregation, and demonstrates a complete Flink‑Hologres pipeline—including table creation, streaming joins, windowed aggregation, and query examples—for fine‑grained user metric analysis.
Suning’s senior architect Yang Zhaohui explains how his team rebuilt the tag platform with ClickHouse, using RoaringBitmap and custom optimizations to achieve second-level queries on billions of user tags, dramatically cutting response time, reducing hardware costs, and enabling real-time marketing insights.
This tutorial demonstrates how to create a VARBINARY test table in HBase using Phoenix, serialize its data with RoaringBitmap, implement a custom Spark aggregation function to merge bitmap values, and query the table via Spark SQL, showcasing a practical big-data processing workflow.
This article describes the evolution of 58.com's user‑set service architecture, the transition from MongoDB to RoaringBitmap storage, and the machine‑learning‑driven algorithm optimizations that enable real‑time, multi‑dimensional, and localized push notifications for millions of users.
