Jul 3, 2024 · Big Data

How to Scale Global Dictionary Indexing with Distributed SQL in Minutes

This article explains a distributed‑computing approach for generating a globally unique integer index from massive string datasets, replacing single‑reducer sorting with hash‑bucket partitioning and parallel processing to cut runtime from 30 minutes to just 2 minutes.

Big DataSQL Optimizationdistributed computing

0 likes · 5 min read

How to Scale Global Dictionary Indexing with Distributed SQL in Minutes

Code Ape Tech Column

Nov 3, 2022 · Databases

Strategies for Splitting Large Keys and Values in Redis to Reduce Memory Usage and Latency

This article explains how to handle Redis scenarios with oversized keys, massive collections, billions of keys, and large bitmaps or Bloom filters by partitioning data into multiple keys or hashes, using bucketization, multi‑get, and careful bitmap splitting to improve performance and lower memory consumption.

BitmapMemory Optimizationbloom-filter

0 likes · 8 min read

Strategies for Splitting Large Keys and Values in Redis to Reduce Memory Usage and Latency