Tagged articles
tiered storage
28 articles · Page 1 of 1
Colibri, a pure C inference engine with zero dependencies, enables running the 744B parameter GLM-5.2 MoE model on consumer hardware with just 25GB RAM and NVMe SSD by leveraging MoE sparsity and a three-tier storage scheduling system across VRAM, RAM, and disk, achieving 0.05–6.8 tok/s depending on hardware.
ColibriGLM-5.2MoE
0 likes · 14 min read

At ten‑million QPS, log data can explode from a few gigabytes to terabytes or even petabytes, triggering storage blow‑up, pipeline saturation, slow queries, runaway costs, and poor signal‑to‑noise, and the article breaks down ingest, index, and store costs while presenting edge sampling, label‑based indexing, tiered storage, and log‑to‑metric rollup as mitigation tactics.
Cost OptimizationElasticsearchLogging
0 likes · 19 min read

This article deep-dives into Apache Fluss architecture, detailing its Master-Worker design, dual LogStore and KvStore engines, tablet-based sharding, tiered hot/cold storage on remote object stores, and Flink-centric client integration, explaining how these components unify streaming and lakehouse workloads.
Apache FlussFlink connectorKvStore
0 likes · 14 min read

The article analyzes why storing massive tracing spans in a single Elasticsearch cluster fails at high QPS, outlines the four key challenges of trace data, and presents a three‑step engineering solution—sampling, hot‑warm‑cold tiered storage, and separating indexes from span payloads—while comparing major back‑ends such as Elasticsearch, Cassandra, ClickHouse, and Tempo.
ClickHouseDistributed TracingElasticsearch
0 likes · 19 min read

China's high-quality dataset count hit 120,000 totaling 1,565 PB with 60% quarterly growth, exposing four systemic gaps in traditional BI-oriented lakehouse architectures — storage, governance, compute performance, and security — and driving an AI-native reference architecture built on Paimon, StarRocks, tiered storage, operator-level lineage, and multi-modal retrieval.
AI training dataApache PaimonStarRocks
0 likes · 43 min read

TiDB’s next‑generation tiered storage separates hot data on local SSDs from cold data on object storage, using multi‑tree LSM, segment caching, and IA Manager to reduce storage costs by up to 50 % while preserving low latency for frequent accesses and providing clear migration and monitoring guidelines.
Cold DataIA ManagerS3
0 likes · 15 min read

Apache Fluss graduates to a top-level project, introducing the Lakestream architecture that unifies real-time streaming and historical lakehouse storage through tiered hot/cold storage, unified metadata, and Union Read, eliminating Lambda architecture complexity and enabling sub-second analytics on a single logical table.
Apache FlussFlinkLakestream
0 likes · 15 min read

This article walks through Milvus 2.3‑2.6.x storage optimizations—Mmap, tiered storage, and clustering compaction—explaining their principles, configuration hierarchy, benchmark results, and concrete deployment templates that together can boost query performance up to 25‑fold while halving memory consumption.
MilvusPerformance Tuningclustering compaction
0 likes · 24 min read

This article details JD Advertising's journey from a 1 PB Apache Doris data lake to a multi‑level hot‑cold tiering architecture, describing two tiering strategies, the performance and schema‑change challenges faced during the upgrade to Doris 2.0, and the optimizations that reduced storage costs by about 87% while boosting query throughput.
Apache DorisCold DataHot Data
0 likes · 19 min read

This article details JD's large‑scale HDFS unified storage implementation, covering cross‑region storage challenges, topology design, asynchronous block replication, flow‑control mechanisms, tiered storage strategies, automatic hot‑cold data migration, and the resulting performance and cost improvements for big‑data workloads.
Cross-Region StorageData ManagementHDFS
0 likes · 20 min read

This article presents JD Retail's large‑scale HDFS deployment, detailing its unified storage architecture, cross‑region data replication challenges and solutions, tiered storage strategies for hot, warm and cold data, and the operational modules that together improve performance, reliability and cost efficiency in a big‑data environment.
Cross-Region StorageHDFSbig data
0 likes · 21 min read

Facing exploding Kafka scale, Xiaohongshu’s data‑storage team adopted a cloud‑native design that introduces tiered hot‑cold storage, containerization, and a custom load‑balancing service, achieving dramatic storage‑cost reductions, minute‑level cluster migrations, high‑performance data access, and automated resource scheduling.
Containerizationautoscalingcloud-native
0 likes · 20 min read

Xiaohongshu's big-data storage team built cloud-native architecture with tiered storage, containerized Kafka, and custom load balancer, cutting storage costs up to 60%, enabling minute‑level elastic migration, improving scaling efficiency tenfold, and boosting performance via caching and batch reads.
Cost OptimizationElastic ScalingKafka
0 likes · 20 min read

This article explores Apache RocketMQ 5.x’s new cloud‑native capabilities—including the Proxy component, gRPC client, Pop consumption model, timer‑based delayed messages, tiered storage, distributed rate‑limiting, and containerized deployment on Tencent Cloud—while outlining architecture changes, practical usage patterns, and future directions.
ContainerizationDistributed Rate LimitingRocketMQ
0 likes · 32 min read

This article presents an in‑depth overview of ByteDance's EB‑scale HDFS, covering its new features, multi‑datacenter architecture, tiered storage implementation, data management services, capacity and fault‑tolerance strategies, as well as practical data‑protection mechanisms and related Q&A.
HDFSbig datadata protection
0 likes · 22 min read

This article outlines how a large‑scale Kafka deployment of over a thousand machines across dozens of clusters was engineered for stability and efficiency through a custom Guardian controller that adds partition‑level throttling, automatic balancing, multi‑tenant isolation, cross‑IDC management, tiered storage, audit capabilities, and fully automated operational workflows.
KafkaOperationsauto-balancing
0 likes · 21 min read

The article details how Bilibili operates over 1,000 Kafka nodes across 20+ clusters, outlines the scalability and stability challenges they faced, and explains the design and implementation of their self‑built Guardian federation controller, partition‑level throttling, automatic balancing, multi‑tenant isolation, tiered storage, audit, and automated ops workflows.
Cluster GovernanceKafkaPartition Balancing
0 likes · 21 min read

Redis is exploring a disk‑backed tiered storage architecture, allowing infrequently used data to move from memory to flash and back, a move aimed at reducing costs, improving scalability, and expanding its popularity among developers and gaming applications.
FlashNoSQLRedis
0 likes · 4 min read

The paper presents a tiered NVMe SSD solution for ad‑ranking retrieval, using zone‑aware hardware, Direct I/O with libaio/IoUring, and an adaptive pipeline scheduler to cut long‑tail read latency and boost throughput, enabling larger‑than‑memory signals and improving ranking accuracy and revenue.
LibaioNVMeSSD
0 likes · 20 min read

This article explains how Apache Pulsar combines messaging, storage, and lightweight function computing into a cloud‑native streaming platform, detailing its architecture, storage‑compute separation, tiered storage, pluggable protocols, reliability guarantees, and rich ecosystem compared with traditional queues and Kafka.
Apache PulsarData Reliabilitycloud-native
0 likes · 10 min read

RocketMQ 5.1.0 introduces a tiered storage module that offloads messages from local disks to cheaper storage, enabling longer retention without impacting hot data performance, and the article explains its design, implementation details, configuration steps, and current challenges.
BackendMessage QueueRocketMQ
0 likes · 12 min read

The article explains how RocketMQ 5.1.0 introduces a tiered storage module that offloads messages to cheaper media, describes its design, architecture layers, quick‑start configuration, upload and read mechanisms, prefetch cache, fault recovery, current development plans, and remaining challenges.
Message QueueRocketMQtiered storage
0 likes · 13 min read

This article presents JD's large‑scale big‑data platform, detailing its overall architecture, the challenges of cross‑region storage, the design of a unified cross‑domain data synchronization mechanism, and the implementation of tiered storage to improve performance, cost efficiency, and data reliability across multi‑datacenter clusters.
HDFSbig datacross region
0 likes · 15 min read

This article explains how tiered storage in cloud‑native data warehouses like AnalyticDB MySQL reduces storage costs while maintaining performance by separating hot and cold data, automating migration, and optimizing cold‑data access through caching and archiving techniques.
AnalyticDB MySQLCost Optimizationcold hot data
0 likes · 15 min read

This article details a three‑part technical sharing that covers cost governance for offline Hadoop clusters, a large‑scale data‑center migration with architecture upgrades, and a tiered storage strategy using EC and COS to reduce storage costs and improve performance in a cloud‑native big‑data environment.
Big Data MigrationCOSErasure Coding
0 likes · 10 min read


ITPUBMar 28, 2019 · Big Data
Pravega, Dell’s native stream storage project, addresses the challenges of modern low‑latency, exactly‑once stream processing by combining tiered storage, Apache BookKeeper, and seamless Flink integration, offering a unified solution that reduces development, storage, and operational costs compared to traditional message systems like Kafka.
Apache FlinkKafka ComparisonPravega
0 likes · 10 min read

The article explains how traditional SAN storage creates CPU‑I/O bottlenecks, how Google’s distributed file system inspired hyper‑converged designs that fuse compute and storage, and why tiered storage with SSD and HDD offers scalable, high‑performance infrastructure for modern data‑center workloads.
I/O performanceStoragedata center
0 likes · 11 min read

This article explains Hadoop’s tiered storage concept, describing how data is classified by temperature—hot, warm, cold, frozen—and automatically moved across disk and archive layers to optimize cost and performance, with examples from Hadoop versions and eBay’s large‑scale deployment.
Data TemperatureHDFSHadoop
0 likes · 9 min read
