This article compares the storage models, data organization, indexing, read/write processes, and performance trade‑offs of three major message queues—Kafka, RocketMQ, and JMQ—providing detailed technical insights for architects and engineers making storage‑related design decisions.
This article analyzes the technical principles, architectural innovations, and real‑world scenarios of Data Processing Units (DPUs), showing how they resolve storage‑CPU mismatches, eliminate excessive east‑west traffic, and accelerate failure recovery, thereby becoming a core infrastructure for AI and cloud computing.
Huolala, founded in 2013, runs a massive cross‑cloud hybrid big‑data storage platform of over 40 PB across 3,000+ machines, evolving through four online‑storage phases, robust HA design, performance‑cost optimizations, AI vector storage, and a cost‑governance system that saved more than half of its storage expenses.
This article explains how databases store data on various storage devices, why indexes dramatically improve query performance through sorted structures and binary search, and outlines practical SQL optimization techniques while warning about the trade‑offs of excessive indexing.
3FS is an open‑source distributed file system that abstracts multiple machines into a single namespace, offering massive scalability, fault tolerance, and high throughput through components like Meta, Mgmtd, Storage, and Client, and leveraging the CRAQ protocol for strong consistency and efficient reads and writes.
This article provides a concise overview of four key distributed file systems—HDFS, CephFS, GFS, and TFS—explaining their architectures, strengths, weaknesses, and typical application scenarios for large‑scale data storage and processing.
RocketMQ tackles the high‑performance, high‑reliability challenges of distributed messaging by combining sequential disk writes, memory caching, and multi‑level indexing, detailing its storage logic, core structures, zero‑copy techniques, replication modes, static topic scaling, and practical tuning guidelines for optimal throughput.
This article explains how databases store data on various storage media, how indexes—especially clustered indexes—leverage sorted structures and binary search to dramatically reduce query time, and outlines best practices and pitfalls of index usage and SQL optimization.
This article provides a detailed, step‑by‑step analysis of RabbitMQ's classic queue storage architecture, covering directory layout, index and data file formats, write and read workflows, file compaction, and practical operational tips such as publisher confirms and manual acknowledgments to improve reliability and throughput.
This article explores the principles of data replication, comparing shared memory, shared disk, and non‑shared storage architectures, detailing replication formats, consistency challenges, and various replication strategies such as synchronous, asynchronous, semi‑synchronous, and majority‑based approaches, helping engineers choose the right trade‑offs.
Learn how to design, configure, and deploy a Ceph distributed storage cluster using ceph‑deploy, covering storage fundamentals, Ceph architecture, component roles, network planning, OS preparation, mon, mgr, osd setup, and dashboard activation, with detailed commands and best‑practice recommendations for production environments.
This article analyzes the evolution of distributed storage—from traditional compute‑storage separation to edge‑centric, AI‑enabled architectures—covering service models, key technologies such as CXL and erasure coding, reliability strategies, performance optimizations, vendor landscapes, and emerging green and intelligent trends.
This article provides a comprehensive overview of Ceph distributed storage, covering storage fundamentals, Ceph architecture, advantages, version lifecycle, and step‑by‑step deployment using ceph‑deploy, including environment preparation, monitor and OSD setup, manager configuration, and dashboard activation.
The article analyzes the challenges of deploying large language models on cloud servers—such as latency, security, and constant connectivity—and explains how near‑memory computing architectures (PNM, PIM, CIM) can integrate storage and processing to enable efficient, high‑performance edge AI deployments, outlining the trade‑offs of each approach.
This article compares 3FS, vePFS, and CloudFS across metadata and data planes, presents detailed benchmark results for AI training scenarios, analyzes architectural trade‑offs, and draws insights for future cloud‑native file storage development.
Bilibili’s comment service, a critical component for user interaction, faces massive read‑write traffic that can overwhelm TiDB, so the team built a multi‑level storage architecture using Redis sorted‑sets for indexes and a custom Taishan KV store, adding automatic degradation, consistency mechanisms, and hedging policies to ensure high availability and performance.
This article presents a comprehensive design of Bilibili's comment service architecture, detailing the transition from TiDB to a multi‑level storage system based on Taishan KV, the data models, consistency mechanisms, retry and versioning strategies, and a hedging‑based degradation policy to ensure high availability under heavy traffic.
The paper proposes a multi‑level storage architecture for Bilibili’s comment service that replaces TiDB with a custom KV store (Taishan) and Redis caching, introduces unstructured indexes, CAS‑based consistency, real‑time and offline reconciliation, and a hedged degradation strategy to boost reliability, read throughput, and scalability during traffic spikes.
Baidu’s Canghai Storage unifies metadata, hierarchical namespace, and data layers into a Meta‑Aware, three‑generation architecture that scales to trillions of metadata items and zettabyte‑scale data, using a distributed transactional KV store, single‑machine‑distributed namespace, and online erasure‑coding micro‑services to deliver high performance, low cost, and seamless scalability.
This article analyzes the technical roadmap for upgrading AI super‑large GPU clusters to support trillion‑parameter multimodal models, covering single‑chip performance, super‑node scaling, DPU‑based compute fusion, energy‑efficient designs, converged storage, high‑throughput networking, and fault‑tolerant checkpoint strategies.
RocketMQ stores messages in a sequential CommitLog while using a ConsumerQueue index for fast random access, and employs two looping threads—RebalanceService for load‑balancing queue assignments and PullMessageService for continuously pulling messages and dispatching them to user callbacks via a thread‑pooled consumer pipeline.
This article thoroughly examines Kafka's storage architecture, analyzing its design motivations, storage mechanisms, log formats, partitioning, indexing, cleanup strategies, and performance optimizations such as sequential log writes, sparse indexing, page cache, and zero‑copy, providing insights applicable to other storage systems.
This article dissects Pulsar’s storage layer by examining the Bookie component, detailing its architecture, the sequential write‑ahead log, ledger management, journal handling, caching mechanisms, and the complete read/write call chains with concrete code examples and performance considerations.
This article traces the two‑decade evolution of message queues—from early decoupling solutions like ActiveMQ, through high‑throughput designs such as Kafka, to modern platformized systems like RocketMQ and Pulsar—while explaining fundamental concepts, partitioning, and storage architectures that underpin today’s distributed messaging platforms.
In his 2023 keynote, Baidu Vice President Xie Guangjun unveiled an AI‑native cloud architecture featuring 7th‑gen G7 servers, Kunlun R300 and Ascend 910B instances, a gateway with X86, programmable switches and FPGA, unified storage (TafDB, Aries, BOS, CDS, PFS), Baige 3.0 AI infrastructure, an intelligent computing network, GaiaDB 4.0, BMR Spark 3.2, SugarBot natural‑language analytics, distributed edge and private cloud, and video and low‑code platforms.
Software-defined storage (SDS) combines distributed and virtualization technologies to decouple storage software from hardware, offering flexible scaling, higher utilization, and a market dominated by a few Chinese vendors, while outlining its architecture, benefits, and competitive landscape.
This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.
This article explains the three core benefits of message queues, compares major MQ products, details RocketMQ's advantages, internal models, reliability mechanisms, load‑balancing, storage design, and practical code examples, guiding engineers to make informed architectural decisions.
This article examines the evolution of big‑data storage from on‑premise HDFS to cloud‑native object storage, compares their architectures and performance, outlines future lakehouse storage requirements, and demonstrates a practical implementation using the JuiceFS distributed file system.
This article provides a comprehensive guide to planning, structuring, securing, and managing a data lake on Azure Data Lake Storage Gen2, covering zone architecture, folder hierarchy, access control, file formats, scalability considerations, and best‑practice recommendations for big‑data workloads.
This article introduces RDF and property graph models, explains the core goals of graph database storage, compares mainstream storage solutions such as array, linked‑list and LSM‑Tree approaches, and presents practical deployment experiences of the Galaxybase distributed graph database.
This article explains RocketMQ's broker storage architecture, detailing the roles of CommitLog, ConsumeQueue, and IndexFile, and describes how messages are formatted, indexed, and written to disk using memory‑mapped files to achieve high throughput and reliability.
This article provides an in‑depth technical analysis of Kafka’s storage architecture, covering its design goals, storage mechanisms, log segment layout, sparse indexing, log cleanup policies, and the performance techniques such as sequential writes, page cache, and zero‑copy that enable high‑throughput streaming.
The article explains core flash storage techniques—including global load balancing, multi‑stream partitioning, and end‑to‑end I/O priority—by describing Active‑Active architectures, hash‑based request distribution, cache mirroring, data classification, and latency management to improve performance, reliability, and SSD lifespan.
The article explains the fully interconnected architecture of an all‑flash storage system, covering redundant FRU modules, RDMA‑based high‑speed networking, intelligent disk enclosures, SSD structure, wear‑leveling, bad‑block management, data redundancy, and the differences between SAS and NVMe protocols.
This article introduces graph database fundamentals, explains why graph databases are needed, outlines core storage goals such as index‑free adjacency, compares array, linked‑list and LSM‑tree storage schemes, and presents the design, performance advantages, and real‑world applications of the Galaxybase distributed graph database.
This article analyzes the storage challenges of high‑performance computing—including traditional HPC, AI‑driven HPC, and high‑performance data analysis—examines Baidu’s internal practices, and presents the Canghai storage platform with its object storage, parallel file system (PFS) and RapidFS solutions that address throughput, latency, and scalability requirements.
The talk outlines IoT data classifications, requirements, and proposes a one‑stop storage product using multi‑model support, columnar formats, compute‑storage separation, tiered storage, and query optimization to achieve ten‑fold cost reduction and ten‑fold performance gains.
FastDFS is a lightweight, open‑source distributed file system written in C that uses a three‑component architecture—client, tracker server for load‑balancing and discovery, and storage servers with push‑based binlog replication—to handle high‑concurrency upload/download of small to medium files, support group‑wide synchronization, optional trunk storage, Nginx anti‑leech integration, and extensible deduplication via FastDHT.
This article explains RocketMQ’s disk‑based storage design—including CommitLog, ConsumeQueue, and Index files—detailing sequential writes, memory‑mapped I/O, flexible flush strategies, and memory‑level read/write separation that together enable unlimited backlog, high throughput, and low latency messaging.
This article provides a comprehensive technical analysis of Apache RocketMQ's storage subsystem, detailing the architecture of CommitLog, ConsumeQueue, and IndexFile, the use of PageCache and mmap, the message write path, flush strategies, and read mechanisms, complete with code excerpts and diagrams.
This article examines Android 6.0’s storage architecture, detailing the roles and interactions of MountService, Vold, and the kernel, illustrating module and process structures, communication flows via sockets, and highlighting performance considerations such as thread blocking and ANR issues.
This article explains the origins and purpose of data lakes, outlines four key construction goals, details common ingestion methods and storage technologies, and describes essential governance practices such as cataloging, data quality, and regulatory compliance.
This article presents a comprehensive design of a cloud‑native ClickHouse OLAP system, detailing its three‑layer architecture, storage‑compute separation, unified metadata management, high‑availability mechanisms, elastic scaling, cost reductions, and future enhancements for multi‑replica and MPP query support.
This article examines the evolving data‑storage demands of large‑scale commercial advertising systems, traces the development of BaikalDB from early MySQL sharding to a heterogeneous, cloud‑native distributed database, and explains its storage, compute, and scheduling designs that deliver high reliability, low cost, and millisecond‑level performance.
This article examines the requirements of commercial advertising systems for data storage, traces the evolution from single‑node MySQL to BaikalDB’s cloud‑native, MySQL‑compatible distributed architecture, and details its storage, compute, and scheduling designs, highlighting key features such as Raft replication, RocksDB storage, and hybrid OLTP/OLAP support.
This article analyzes the evolution of Baidu's advertising data storage, detailing the business-driven requirements, the design and development of the BaikalDB distributed database, its architecture across storage, compute and scheduling layers, key features such as Raft replication and multi‑index support, and the lessons learned for building cloud‑native, high‑performance databases.
This article explores the evolution and current landscape of time‑series databases, detailing design principles, storage structures such as B‑Tree, B+Tree, and LSM‑Tree, and providing an in‑depth comparison of OpenTSDB, InfluxDB, and the emerging Apache IoTDB, while also discussing practical deployment considerations and industry use cases.
This article examines the core concepts, requirements, architectural models, persistence strategies, scalability, high‑availability mechanisms, performance optimizations, security models, and practical considerations of distributed file systems such as HDFS, GFS, and Ceph, offering a comprehensive guide for engineers and researchers.
This article explains the evolution of message‑queue middleware by comparing the basic double‑ended queue implementation, Redis list usage, Kafka’s partitioned log architecture with segment files and sparse indexes, and Pulsar’s compute‑storage separation using BookKeeper, highlighting their designs, strengths, and trade‑offs.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements, architectural models with and without central nodes, persistence strategies, scalability, high availability, performance optimization, security mechanisms, and additional considerations such as space allocation, file deletion, small‑file handling, and deduplication.
This article introduces Intel® Optane™ Persistent Memory (PMem), explains its performance characteristics, usage modes, the SNIA NVM programming model, and details the PMDK libraries and related tools that enable developers to build high‑performance, fault‑tolerant applications on persistent memory.
This article explores the core concepts, design requirements, architectural models, scalability, high availability, performance optimization, and security considerations of distributed file systems, comparing centralized and decentralized approaches while highlighting practical solutions for persistence, consistency, and fault tolerance.
This article explains the types, components, operation principles, data layout, read/write process, locality-based prefetching, fragmentation, capacity calculations, and RAID configurations of modern hard disk drives, providing a comprehensive technical overview for anyone interested in storage fundamentals.
This article compares the fundamentals and limitations of Redis list‑based queues with Kafka and Pulsar, explaining how partitions, cursors, replication, segment storage, and consumption models overcome Redis’s drawbacks and enable high‑performance, scalable, and highly available messaging systems.
The article explains the concept and advantages of data lakes, outlines the major storage and acceleration challenges they face, provides a checklist for ideal data‑lake solutions, and details how Alibaba Cloud's JindoFS addresses those challenges with object‑storage‑based, high‑performance, scalable features.
This article provides a comprehensive overview of Red Hat Ceph’s distributed object‑storage architecture, covering storage pools, authentication, placement groups, the CRUSH algorithm, replication, erasure coding, internal operations, high‑availability mechanisms, client interfaces, and encryption, illustrated with diagrams and practical details.
The article provides a comprehensive overview of the Lustre parallel file system architecture, its core components, POSIX compliance, scalability, high‑performance networking, security features, data layout mechanisms, and performance considerations for large and small files, along with practical optimization tips for HPC environments.
This article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its historical development, architectural models—including SAN, NSD, and Share‑Nothing Cluster—its operational capabilities, performance advantages, scalability, high‑availability features, and its role in large‑scale high‑performance computing environments.
WeChat Pay’s Hermes‑based log system ingests trillions of entries daily, storing petabytes across a 200‑node HDFS cluster with four‑nine availability, while LSM‑style writes, separate inverted indexes and hot‑cold tiering cut memory, disk and cost by up to 70 % and keep 95 % of queries under five seconds.
Data deduplication eliminates redundant data blocks to lower storage and bandwidth costs, while source‑ and transmission‑level encryption safeguards data in transit and at rest; the article also compares hardware vs software deduplication, various storage architectures (DAS, SAN, NAS, object and distributed storage) and their trade‑offs.
This article presents a comprehensive design and deployment plan for a bank's unstructured data service platform, covering data growth challenges, lifecycle management, three‑tier storage architecture, Elasticsearch indexing, fault‑tolerant disaster recovery, monitoring, and future development directions.
This article examines the evolution, core requirements, architectural models (centralized and decentralized), persistence strategies, scalability, high availability, performance optimization, security mechanisms, and additional design trade‑offs of distributed file systems, providing a comprehensive overview for architects and engineers.
The article thoroughly examines the Linux SCSI subsystem architecture, illustrating how abstract hardware topology—hosts, channels, IDs, and LUNs—is modeled in software through structs such as scsi_host and scsi_device, and detailing initialization, scanning, path establishment, power management, and block‑layer integration to guide kernel storage driver design.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential design requirements, centralized and decentralized architecture models, persistence, scalability, high availability, performance optimization, security, and additional practical aspects such as space allocation, file deletion, small‑file handling, and deduplication.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements such as POSIX compliance, persistence, scalability, and security, and compares centralized (e.g., GFS) and decentralized (e.g., Ceph) architectures, along with strategies for high availability, performance optimization, and handling small files.
This guide explains how to plan Ceph hardware by balancing CPU, memory, storage, network bandwidth, and failure domains, offering practical recommendations for daemons, OSDs, monitors, managers, and SSD vs HDD choices to achieve cost‑effective, high‑performance large‑scale clusters.
This article explains the fundamental architecture of SSDs, covering their NAND‑flash composition, logical sectors, page‑level operations, flash cell types (SLC, MLC, TLC), and how data is distributed across multiple dies to achieve performance and capacity.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements, architectural models with and without a central node, persistence strategies, scalability, high availability, performance optimizations, security mechanisms, and practical considerations for small‑file workloads.
eMMC is an embedded flash storage standard defined by JEDEC that integrates NAND flash, a controller with FTL, and an 8‑bit parallel interface, offering small size, low power, and high capacity, with defined working modes, six configuration registers, and performance advantages over raw NAND, MMC, and UFS.
This article provides a comprehensive overview of distributed file systems, covering their historical evolution, essential requirements such as POSIX compliance, persistence, scalability and security, architectural models with and without a central node, replication strategies, load balancing, high availability, performance optimizations, and practical considerations for small‑file handling.
This article explains how UCloud's performance‑oriented UFS leverages NVMe SSDs, upgrades to NFSv4, redesigns business indexing, and introduces a novel append‑only storage engine with stream/extent architecture to achieve sub‑10 ms latency and high IOPS for demanding AI and analytics workloads.
This article provides a comprehensive overview of HDFS, covering its purpose, architecture, read/write processes, component roles, command-line tools, replica placement strategies, and the advantages and disadvantages of using Hadoop's distributed file system for large-scale data storage.
The article explains the MFS distributed file system, detailing its four components—Master, Metalogger, Chunkserver, and Client—along with hardware recommendations, metadata handling, replication strategies, and FUSE‑based client mounting, providing a comprehensive guide to building a GoogleFS‑like storage cluster.
This article provides a comprehensive technical overview of IBM GPFS (General Parallel File System), detailing its core components, cluster management roles, networking models, and best‑practice building‑block configurations for high‑performance computing environments.
This article explains the NVMe (Non‑Volatile Memory Express) standard, its logical device interface, key attributes, queue architecture, namespace concepts, multi‑path I/O, SR‑IOV support, and how it compares to traditional SCSI storage, providing a comprehensive technical overview for modern data‑center and client systems.
Alibaba introduces Open Channel SSD and Fusion Engine, a unified storage architecture that addresses modern data‑center challenges by enhancing performance, flexibility, cost efficiency, and determinism through open hardware interfaces and advanced user‑space storage engines.
This article explains the fundamentals of NVMe‑oF, the role of InfiniBand components such as HCAs, switches, subnet managers and gateways, and why NetApp chose InfiniBand for its EF570/E5700 storage systems, highlighting performance benefits and protocol coexistence.
The explosive growth of big‑data applications is forcing storage vendors to redesign architectures for petabyte‑scale capacity, real‑time latency, high IOPS, security compliance, and cost efficiency, while also addressing flexibility, data longevity, and the needs of both large and small users.
The article summarizes PureStorage’s Matt Kixmoeller’s blog, which critiques Dell EMC’s PowerMax as a rebranded, partially NVMe‑optimized VMAX, highlights Pure’s own NVMe‑centric strategy, points out performance and efficiency shortcomings, and discusses the broader impact of NVMe on enterprise storage design.
This article provides a comprehensive overview of IBM GPFS, covering its architecture, management components, networking models, cluster and storage design, as well as practical guidance on building‑block configurations for performance and capacity scaling in high‑performance computing environments.
With data volumes soaring to petabyte scales, the article explains why object storage is the only viable solution for massive storage needs, outlines procurement considerations, design principles, and operational challenges, and offers practical guidance for building, evaluating, and scaling PB‑level storage systems.
Distributed file systems extend traditional local storage by partitioning data across multiple servers, using a master node for metadata and coordination, handling namespace, replication, load balancing, caching, and client interfaces, thereby overcoming file size, quantity, and concurrency constraints of ext3, reiserfs, and similar local filesystems.
The article analyzes HPE’s purchase of Nimble Storage, detailing the company’s flash‑focused product portfolio, underlying CASL architecture, all‑flash AF series specifications, multicloud integration, and the broader market dynamics affecting traditional and startup storage vendors.
This article shares a series of Sage Weil’s Ceph video lectures that thoroughly explain Ceph’s architecture, design philosophy, key features, and the BlueStore storage engine, providing links to additional resources and background on Ceph’s development history for enthusiasts seeking an in‑depth understanding.
The article explains how traditional centralized backup indexes become performance bottlenecks as data grows, and details Simpana's two‑level distributed indexing architecture—primary and secondary indexes—showing how it enhances backup speed, reduces network load, and simplifies recovery across multi‑site environments.
This article provides a comprehensive overview of flash memory technologies, comparing NOR and NAND flash, explaining cell structures such as SLC, MLC, and TLC, detailing NAND flash architecture, SSD controller functions, wear‑leveling, over‑provisioning, and emerging trends like NVMe and 3D‑Flash.
This article provides an in‑depth technical overview of Meizu's cloud sync service, covering its business scenarios, the evolution of its core SyncML‑based protocols, modular architecture across multiple data centers, performance optimizations such as traffic reduction and connection handling, as well as high‑availability, security, and future SDK‑API extensions.
The article provides a comprehensive overview of IBM's General Parallel File System (GPFS), detailing its physical and logical architecture, key components such as NSD and quorum mechanisms, scalability, load balancing, and fault‑tolerance features for parallel and serial applications.
This article describes the design, architecture, and key features of a Linux‑based distributed file system that emphasizes horizontal scalability, data consistency, fault tolerance, and efficient handling of both small and large files without traditional metadata services.
Kafka persists messages on disk by organizing each topic into multiple partitions, which are further divided into segment files containing paired .index and .log files; this structure enables efficient storage, offset-based lookup, and fast retrieval of specific messages through binary search across segment indexes.
Taobao’s File System (TFS) is a highly available, high‑performance distributed storage solution built on Linux servers, featuring name‑server and data‑server clusters, block‑level replication, HA mechanisms, client caching, seamless scaling, multi‑data‑center disaster recovery, and open‑source support for C++, Java, and Nginx integration.
The article explains FastDFS, an open‑source lightweight distributed file system built from tracker servers, storage servers, and client libraries, detailing its component roles, file upload/download workflows, storage organization, small‑file merging, HTTP support, and current limitations such as data safety and load balancing.
