0 views collected around this technical thread.
This article explains how a CDP system tackles the storage and set‑operation challenges of billions of user‑ID tags and groups by adopting bitmap techniques, especially RoaringBitmap, to dramatically reduce space usage and enable fast union, intersection, and difference calculations.
This article examines how bitmap data structures and multithreading can dramatically accelerate blacklist filtering in large‑scale marketing systems, reducing processing time from tens of minutes to milliseconds while saving memory and improving overall system performance.
This article explores how to achieve fast, scalable data retrieval in million‑level high‑concurrency scenarios by replacing naïve full‑combination rule matching with column‑wise inverted indexes and bitmap operations, dramatically reducing time complexity and improving stability while leveraging RoaringBitmap compression for space efficiency.
The article explains how Hologres RoaringBitmap compresses user ID sets into efficient bitmap indexes, splits 64‑bit IDs into buckets, syncs them from MaxCompute, and enables sub‑second user portrait queries that previously took minutes, dramatically improving performance and scalability.
RoaringBitmap improves traditional BitMap by lazily allocating four container types, compressing sparse data, and dynamically switching between array, bitmap, and run containers, enabling fast exact set operations that power big‑data systems such as Kylin, ClickHouse, and B‑Station’s user‑visit and crowd‑package pipelines, dramatically reducing memory use and processing latency.
By representing pre‑and post‑refresh SPU sets as RoaringBitmaps and diffing them, the system avoids full‑insert writes, cuts memory usage by orders of magnitude, speeds refreshes by over 50 % and reduces write volume nearly 87 %, solving large‑scale tag‑based product refresh challenges.
For deduplicating game recommendations in Redis, RoaringBitMap outperforms intset and Bloom filters by storing 300 auto‑incrementing game IDs in roughly 0.5 KB—over twice the compression of intset and far smaller than the 29 KB Bloom filter—thereby cutting memory use, latency, and hardware costs.
This article explains how to implement both offline (T+1) and real‑time UV counting using Hologres with RoaringBitmap for high‑cardinality aggregation, and demonstrates a complete Flink‑Hologres pipeline—including table creation, streaming joins, windowed aggregation, and query examples—for fine‑grained user metric analysis.
This article describes the evolution of 58.com's user‑set service architecture, the transition from MongoDB to RoaringBitmap storage, and the machine‑learning‑driven algorithm optimizations that enable real‑time, multi‑dimensional, and localized push notifications for millions of users.