Large Scale Storage

DataFunSummit

Aug 24, 2024 · Databases

Cloud‑Native Storage Solutions for Large‑Scale Vector Data with Milvus and Zilliz

This article presents a comprehensive overview of Zilliz’s cloud‑native vector database ecosystem, detailing Milvus’s distributed architecture, indexing and query capabilities, related tools such as Towhee and GPTCache, storage challenges, tiered storage designs, performance metrics, and real‑world AI use cases like code‑assist and RAG‑based Q&A systems.

AI infrastructureANN SearchLarge Scale Storage

0 likes · 21 min read

Cloud‑Native Storage Solutions for Large‑Scale Vector Data with Milvus and Zilliz

Efficient Ops

May 11, 2021 · Big Data

How to Store Billions of Keys in Redis: Cut Memory, Reduce Fragmentation, and Scale Real‑Time DMP

This article examines the challenges of storing massive DMP data in Redis, analyzes memory fragmentation, key‑size issues, and latency constraints, and presents practical strategies such as TTL eviction, bucket‑hashing, custom key compression, and fragmentation‑reduction techniques to enable scalable, real‑time querying.

BucketIdDMPHashing

0 likes · 11 min read

How to Store Billions of Keys in Redis: Cut Memory, Reduce Fragmentation, and Scale Real‑Time DMP

Architect

Apr 30, 2021 · Backend Development

Designing a Scalable Real‑Time Data Warehouse with Redis: Challenges and Solutions

The article analyzes the massive storage and performance challenges of a real‑time DMP cache built on Redis, outlines data characteristics and technical obstacles, and proposes eviction policies, bucket‑based hashing, and fragmentation‑reduction techniques with Java code examples to achieve billion‑scale in‑memory key‑value storage.

JavaKey-Value StoreLarge Scale Storage

0 likes · 10 min read

Designing a Scalable Real‑Time Data Warehouse with Redis: Challenges and Solutions

360 Zhihui Cloud Developer

Nov 28, 2017 · Databases

Inside Pika: How 360’s Large‑Scale Redis‑Compatible Store Powers Billions of Requests

The article recaps the 2017 Pika community salon in Beijing, explains Pika’s design as a high‑capacity Redis‑compatible storage, showcases real‑world deployments at 360, Huanxin and Weibo, and shares insights from four technical talks along with event highlights and resources.

Large Scale StoragePikaRedis-compatible

0 likes · 9 min read

Inside Pika: How 360’s Large‑Scale Redis‑Compatible Store Powers Billions of Requests

360 Zhihui Cloud Developer

May 23, 2017 · Databases

Why Pika Is the Cost‑Effective, Large‑Scale Alternative to Redis

Pika, developed by 360’s Web Platform and DBA teams, is a Redis‑compatible, disk‑based key‑value store that addresses Redis’s high memory cost and scalability limits by offering persistent storage, multi‑threading, compression, and easy migration tools, making large‑capacity data handling cheaper and more reliable for enterprises.

DatabaseDisk-Based DBKey-Value Store

0 likes · 23 min read

Why Pika Is the Cost‑Effective, Large‑Scale Alternative to Redis