Colibri, a pure C inference engine with zero dependencies, enables running the 744B parameter GLM-5.2 MoE model on consumer hardware with just 25GB RAM and NVMe SSD by leveraging MoE sparsity and a three-tier storage scheduling system across VRAM, RAM, and disk, achieving 0.05–6.8 tok/s depending on hardware.
At ten‑million QPS, log data can explode from a few gigabytes to terabytes or even petabytes, triggering storage blow‑up, pipeline saturation, slow queries, runaway costs, and poor signal‑to‑noise, and the article breaks down ingest, index, and store costs while presenting edge sampling, label‑based indexing, tiered storage, and log‑to‑metric rollup as mitigation tactics.
This article deep-dives into Apache Fluss architecture, detailing its Master-Worker design, dual LogStore and KvStore engines, tablet-based sharding, tiered hot/cold storage on remote object stores, and Flink-centric client integration, explaining how these components unify streaming and lakehouse workloads.
The article analyzes why storing massive tracing spans in a single Elasticsearch cluster fails at high QPS, outlines the four key challenges of trace data, and presents a three‑step engineering solution—sampling, hot‑warm‑cold tiered storage, and separating indexes from span payloads—while comparing major back‑ends such as Elasticsearch, Cassandra, ClickHouse, and Tempo.
China's high-quality dataset count hit 120,000 totaling 1,565 PB with 60% quarterly growth, exposing four systemic gaps in traditional BI-oriented lakehouse architectures — storage, governance, compute performance, and security — and driving an AI-native reference architecture built on Paimon, StarRocks, tiered storage, operator-level lineage, and multi-modal retrieval.
TiDB’s next‑generation tiered storage separates hot data on local SSDs from cold data on object storage, using multi‑tree LSM, segment caching, and IA Manager to reduce storage costs by up to 50 % while preserving low latency for frequent accesses and providing clear migration and monitoring guidelines.
Apache Fluss graduates to a top-level project, introducing the Lakestream architecture that unifies real-time streaming and historical lakehouse storage through tiered hot/cold storage, unified metadata, and Union Read, eliminating Lambda architecture complexity and enabling sub-second analytics on a single logical table.
This article walks through Milvus 2.3‑2.6.x storage optimizations—Mmap, tiered storage, and clustering compaction—explaining their principles, configuration hierarchy, benchmark results, and concrete deployment templates that together can boost query performance up to 25‑fold while halving memory consumption.
This article details JD Advertising's journey from a 1 PB Apache Doris data lake to a multi‑level hot‑cold tiering architecture, describing two tiering strategies, the performance and schema‑change challenges faced during the upgrade to Doris 2.0, and the optimizations that reduced storage costs by about 87% while boosting query throughput.
This article details JD's large‑scale HDFS unified storage implementation, covering cross‑region storage challenges, topology design, asynchronous block replication, flow‑control mechanisms, tiered storage strategies, automatic hot‑cold data migration, and the resulting performance and cost improvements for big‑data workloads.
This article presents JD Retail's large‑scale HDFS deployment, detailing its unified storage architecture, cross‑region data replication challenges and solutions, tiered storage strategies for hot, warm and cold data, and the operational modules that together improve performance, reliability and cost efficiency in a big‑data environment.
Facing exploding Kafka scale, Xiaohongshu’s data‑storage team adopted a cloud‑native design that introduces tiered hot‑cold storage, containerization, and a custom load‑balancing service, achieving dramatic storage‑cost reductions, minute‑level cluster migrations, high‑performance data access, and automated resource scheduling.
Xiaohongshu's big-data storage team built cloud-native architecture with tiered storage, containerized Kafka, and custom load balancer, cutting storage costs up to 60%, enabling minute‑level elastic migration, improving scaling efficiency tenfold, and boosting performance via caching and batch reads.
This article explores Apache RocketMQ 5.x’s new cloud‑native capabilities—including the Proxy component, gRPC client, Pop consumption model, timer‑based delayed messages, tiered storage, distributed rate‑limiting, and containerized deployment on Tencent Cloud—while outlining architecture changes, practical usage patterns, and future directions.
This article presents an in‑depth overview of ByteDance's EB‑scale HDFS, covering its new features, multi‑datacenter architecture, tiered storage implementation, data management services, capacity and fault‑tolerance strategies, as well as practical data‑protection mechanisms and related Q&A.
This article outlines how a large‑scale Kafka deployment of over a thousand machines across dozens of clusters was engineered for stability and efficiency through a custom Guardian controller that adds partition‑level throttling, automatic balancing, multi‑tenant isolation, cross‑IDC management, tiered storage, audit capabilities, and fully automated operational workflows.
The article details how Bilibili operates over 1,000 Kafka nodes across 20+ clusters, outlines the scalability and stability challenges they faced, and explains the design and implementation of their self‑built Guardian federation controller, partition‑level throttling, automatic balancing, multi‑tenant isolation, tiered storage, audit, and automated ops workflows.
Redis is exploring a disk‑backed tiered storage architecture, allowing infrequently used data to move from memory to flash and back, a move aimed at reducing costs, improving scalability, and expanding its popularity among developers and gaming applications.
The paper presents a tiered NVMe SSD solution for ad‑ranking retrieval, using zone‑aware hardware, Direct I/O with libaio/IoUring, and an adaptive pipeline scheduler to cut long‑tail read latency and boost throughput, enabling larger‑than‑memory signals and improving ranking accuracy and revenue.
This article explains how Apache Pulsar combines messaging, storage, and lightweight function computing into a cloud‑native streaming platform, detailing its architecture, storage‑compute separation, tiered storage, pluggable protocols, reliability guarantees, and rich ecosystem compared with traditional queues and Kafka.
RocketMQ 5.1.0 introduces a tiered storage module that offloads messages from local disks to cheaper storage, enabling longer retention without impacting hot data performance, and the article explains its design, implementation details, configuration steps, and current challenges.
The article explains how RocketMQ 5.1.0 introduces a tiered storage module that offloads messages to cheaper media, describes its design, architecture layers, quick‑start configuration, upload and read mechanisms, prefetch cache, fault recovery, current development plans, and remaining challenges.
This article presents JD's large‑scale big‑data platform, detailing its overall architecture, the challenges of cross‑region storage, the design of a unified cross‑domain data synchronization mechanism, and the implementation of tiered storage to improve performance, cost efficiency, and data reliability across multi‑datacenter clusters.
This article explains how tiered storage in cloud‑native data warehouses like AnalyticDB MySQL reduces storage costs while maintaining performance by separating hot and cold data, automating migration, and optimizing cold‑data access through caching and archiving techniques.
This article details a three‑part technical sharing that covers cost governance for offline Hadoop clusters, a large‑scale data‑center migration with architecture upgrades, and a tiered storage strategy using EC and COS to reduce storage costs and improve performance in a cloud‑native big‑data environment.
Pravega, Dell’s native stream storage project, addresses the challenges of modern low‑latency, exactly‑once stream processing by combining tiered storage, Apache BookKeeper, and seamless Flink integration, offering a unified solution that reduces development, storage, and operational costs compared to traditional message systems like Kafka.
The article explains how traditional SAN storage creates CPU‑I/O bottlenecks, how Google’s distributed file system inspired hyper‑converged designs that fuse compute and storage, and why tiered storage with SSD and HDD offers scalable, high‑performance infrastructure for modern data‑center workloads.
This article explains Hadoop’s tiered storage concept, describing how data is classified by temperature—hot, warm, cold, frozen—and automatically moved across disk and archive layers to optimize cost and performance, with examples from Hadoop versions and eBay’s large‑scale deployment.
