This article explains how Kafka partitions data into segments, stores each segment as paired index and log files, and uses sparse indexing to enable efficient queries, illustrating the process with examples and diagrams of segment layout and offset lookup.
This article explains the concept of MySQL compressed tables, highlights strong compression products, discusses why and when to use them, outlines their advantages and drawbacks, details supported compression algorithms, and provides step‑by‑step examples for both MyISAM and InnoDB engines along with their impact on B‑tree pages and the InnoDB buffer pool.
The article explains how to achieve Kubernetes high availability by designing a three‑node control‑plane with stacked etcd, using pod anti‑affinity, tuning node‑monitor timers, handling stale endpoints, configuring TCP keep‑alive, managing node taints and eviction, and choosing RWX storage or appropriate StatefulSet strategies to minimize service disruption after node failures.
The article presents a detailed benchmark of ScaleFlux CSD 2000 storage against Intel SSDs using Percona Server for MySQL 8.0.19 and sysbench, showing how built‑in compression and atomic‑write features give ScaleFlux a clear advantage in large, compressible data sets and high‑concurrency scenarios.
The article analyzes the growing limitations of tape backup, outlines a step‑by‑step migration to disk‑based backup using deduplication, compression and modern storage technologies, and explains how this transition improves reliability, cost efficiency and recovery speed for enterprises.
This step‑by‑step guide shows how to pull a FastDFS Docker image, create tracker and storage containers, adjust their configurations, and verify file upload, providing a fast way to set up the distributed file system on a Mac host.
This article analyzes the core design goals, architectural models, scalability, high‑availability, consistency, security, and performance considerations of modern distributed file systems, comparing classic solutions like NFS and GFS with newer approaches such as Ceph.
This article surveys the rapidly evolving big data landscape by reviewing a wide range of Apache projects—including Hadoop, Spark, Flink, HBase, Kudu, Impala, Kafka, and others—detailing their core components, architectures, strengths, and typical use‑cases for building distributed data platforms.
Distributed locks ensure exclusive access to shared resources across multiple machines, and this article explains their evolution from single-machine locks, classifies system designs, and details Alibaba Cloud Storage’s practical implementation, covering strict mutual exclusion, availability, and lock-switching efficiency with real-world examples.
This comprehensive guide explains why Kubernetes has become the standard container platform, covering its application‑centric architecture, migration challenges, networking and CNI design, control‑plane components, storage solutions including CSI and FlexVolume, image distribution strategies, upgrade and backup procedures, operator patterns, and practical containerization best practices for large‑scale deployments.
This article provides a comprehensive overview of storage fundamentals, covering RAID configurations, COW/ROW snapshot mechanisms, backup strategies, continuous data protection (CDP), clustering concepts, and virtual tape library (VTL) technologies, offering essential knowledge for IT professionals and system architects.
This article systematically explores Git’s underlying architecture, detailing the lifecycle of objects, core data structures, packfile formats, indexing mechanisms, and the algorithms used for object retrieval, while providing practical command examples and visual diagrams to demystify Git’s storage and retrieval models.
This article details how Tongcheng‑eLong transitioned from Docker‑Host deployments to a Kubernetes‑based platform for hundreds of storage and compute services, covering network integration, IP management, service synchronization, storage strategies, operator development, monitoring, logging, and the challenges and future plans they encountered.
This quick reference sheet introduces Laravel's Storage facade, explaining its role as a file handling helper and providing concise examples of common operations such as writing, reading, downloading, managing visibility, and manipulating directories and files using the Storage API.
This article presents a cloud‑native solution for MySQL containerization on Kubernetes using CEPH RBD as backend storage, detailing the background, feasibility analysis with performance benchmarks, the operator‑based architecture, and the benefits for development‑test environments.
This guide walks through setting up a Laravel 5.6 project, installing the Alibaba Cloud OSS storage package, configuring access credentials, creating a custom upload controller, and testing the image upload API with Postman, providing complete code snippets and step‑by‑step instructions.
This article explains common IO latency problems, introduces how to define and refine IO models, lists disk and network evaluation tools, describes key monitoring metrics, and provides practical tuning methods and case studies for improving storage and network performance.
This article explains why mechanical hard drives are slower and less reliable than electronic components, introduces RAID 0, RAID 1, and RAID 5 as solutions to improve speed and fault tolerance, discusses RAID card caching, and shows practical Linux commands for examining RAID hardware and configuration.
In a live Q&A, OPPO cloud platform senior backend engineer shares practical insights on hybrid deployment of offline jobs on Kubernetes, custom high‑performance monitoring with a distributed TSDB, storage scalability via CSI, pod readiness detection, and deploying Ceph/HDFS within the container cluster.
This article provides a comprehensive overview of backup technologies, comparing Host, LAN, LAN-free, Server-free, and Server-less methods, reviews major backup software from EMC, Symantec, Commvault, and IBM, and explains NDMP protocols and NAS backup architectures.
The Linux block layer, positioned between the VFS and device drivers, evolved from a single‑queue design to a multi‑queue architecture (kernel 3.13 onward) to reduce lock contention and exploit modern hardware queues, managing bio/request lifecycles, dispatching through layered queues, and employing various I/O schedulers for balanced throughput and latency.
RAID (Redundant Array of Independent Disks) combines multiple physical drives into logical arrays, offering various levels—such as RAID 0, 1, 5, 6, 7, and 53—that trade off performance, redundancy, and cost, with each level’s architecture, advantages, and drawbacks explained in detail.
This article provides a comprehensive overview of backup technologies, describing the evolution of backup architectures such as Host, LAN, LAN‑free, Server‑free, and Server‑less, and analyzes the features, advantages, and drawbacks of leading enterprise backup software solutions.
Over the past decade, mobile storage has evolved from eMMC to high‑speed UFS, and this article explains UFS’s history, the MPHY physical layer, Unipro protocol stack, UFSHCI host controller, initialization sequence, and data‑transfer pathways, highlighting key registers, lane configurations, and performance metrics.
This article provides a comprehensive guide to Kubernetes storage architecture, covering persistent volumes, claims, the roles of PV, AD, and Volume controllers, the FlexVolume plugin system, and the CSI framework with deployment and usage examples.
The article explains the origins, metrics, and test suites of TPC and SPEC benchmarks, describes their various models for CPU, web, HPC and storage performance, shows how to query official results, and notes a promotional bundle of technical e‑books.
This article describes a real‑world Ceph storage incident caused by disk bad sectors, analyzes its impact, and presents two practical recovery methods—full disk copy with dd+nc and skipping the faulty sector during service start—along with detailed commands and post‑recovery steps.
This article provides a comprehensive overview of snapshot technology, explaining its definition, the backup challenges it addresses, the creation process, various implementations across file systems, LVM, NAS, disk arrays, virtualization, databases, and different snapshot types such as COW, ROW, incremental and continuous data protection, as well as consistency issues and solutions.
This article explains the purpose, new features, API changes, deployment steps, usage examples, limitations, and future roadmap of Kubernetes volume snapshot functionality now in beta as of version 1.17, including required components and supported CSI drivers.
This article explains why Kubernetes is moving in‑tree storage plugins to the Container Storage Interface (CSI), describes the migration process, provides practical commands and YAML examples, and outlines current limitations, roadmap, and recommended actions for users and administrators.
This article explains how SSDs have transformed storage performance and details the characteristics, advantages, and use‑cases of the five main SSD interfaces—SATA, PCIe, M.2, mSATA, and U.2—helping readers choose the appropriate technology for their needs.
This article introduces the motivations for using a distributed file system, explains the architecture and core concepts of FastDFS—including tracker, storage, client, and group—covers its upload and download mechanisms, synchronization management, and the design of its file identifiers, providing a comprehensive overview for developers.
The article explains the differences between cache and buffer, how Oracle’s log buffer and data buffer cache interact with various storage‑level caches, and why write‑back, battery‑backed caches, RAID and disk cache settings are crucial for maintaining data consistency during power loss.
This article chronicles the evolution of flash memory from its 1960s origins with floating‑gate MOSFETs through successive breakthroughs in EPROM, EEPROM, NAND, 3D NAND, and modern SSD technologies, highlighting key inventions, companies, and market milestones over five decades.
This tutorial walks through installing Rook as a cloud‑native storage orchestrator on Kubernetes, deploying the Rook operator, creating a CephCluster CRD, verifying OSD and monitor pods, setting up the Rook toolbox, exposing the Ceph dashboard via a NodePort service, and configuring monitoring and access credentials.
Backend developer Xiao Ming designs an architecture for a photo‑ID app by identifying complexity in image storage and processing, prioritizing solutions such as local‑disk abstraction and Baidu AI services, and applying the principles of suitability, simplicity, and evolutionary design to produce a practical system.
The article explains the concept of Storage Class Memory (SCM), its role as a fast, non‑volatile memory tier between DRAM and NAND flash, and how NVMe and related standards enable new storage hierarchies that improve performance for AI, big data, and emerging technologies.
This article explains the fundamentals, data model, storage options, synchronization strategies, metadata handling, and scaling considerations for building a high‑performance feed‑stream system used in social platforms such as micro‑blogs, friend circles, and short‑video feeds.
The article details Youku’s new media‑asset platform, which replaces a fragmented MySQL‑based system with a domain‑driven entity model stored in Ali‑HBase, leverages Elasticsearch for flexible front‑and back‑end search, and adds an aggregation layer that unifies diverse data sources and reusable computation tasks, delivering high‑availability, low‑latency service for billions of daily API calls.
This guide explains how to safely remove an SSD that was mistakenly added as a data disk in MooseFS by unmounting the MFS directory, editing the mfshdd configuration, restarting the chunkserver, cleaning up fstab entries, and verifying the removal.
This article analyzes the technical distinctions between SLC and MLC NAND flash—covering bit density, cost, device and system performance, endurance, error rates, and enterprise‑grade variants—to explain why SLC remains the preferred choice for demanding industrial environments.
This article provides a comprehensive technical overview of NVMe over TCP, covering its specification status, required host components, namespace limits, latency impact, OS support, performance considerations with DPDK, TCP congestion control, R2T handling, upgrade procedures, open‑source implementations, comparisons with iSCSI, NVMe/FC, and NVMe/RDMA, as well as practical guidance for data‑center deployments.
The article provides a comprehensive overview of NVMe over Fabrics, detailing its RDMA and TCP transport options, NVM subsystem architecture, namespace sharing, multi‑path I/O, reservation mechanisms, and industry adoption, highlighting its performance benefits for AI, ML, and real‑time analytics workloads.
This article explains the NVMe over Fabrics architecture, compares various fabric transports such as FC, InfiniBand, RoCE v2, iWARP and TCP, and details how RDMA‑based technologies like zero‑copy, kernel bypass and CPU‑free transfers give NVMe‑oF its performance advantages while also covering protocol differences, FC‑NVMe, and the emergence of NVMe/TCP.
This article explains TDengine’s architecture, including its management, data, and client modules, virtual node design, write process, and detailed storage file structures, highlighting how its innovative design optimizes resource usage and performance for IoT and other big‑data applications.
BeeGFS, an open‑source parallel file system originally developed by Fraunhofer, has emerged as a flexible, high‑performance alternative to GPFS and Lustre in HPC, driven by growing demands from large‑scale analytics, AI, and cloud storage, with expanding global adoption and ecosystem partnerships.
This article examines how integrating Non‑volatile Memory (NVM) with Redis can replace traditional RDB/AOF persistence, offering a hybrid design that improves durability, reduces recovery time, and maintains high write performance, while discussing implementation details, performance benchmarks, and future optimization challenges.
This article explains how to design a high‑performance feed‑stream architecture—including product definition, data modeling, storage choices, synchronization modes, metadata handling, commenting, likes, sorting, search, and deletion—so that a system can support tens of millions to billions of users while remaining reliable and scalable.
This article explains how incorporating hosted‑cloud physical servers into UCloud's UK8S cluster enables seamless hybrid‑cloud operation by improving resource utilization during low‑traffic periods, simplifying network, storage, and load‑balancing configurations, and reducing the operational overhead of managing separate Kubernetes clusters.
This article explains the principles, advantages, and drawbacks of various RAID configurations—including RAID 0, 1, 10/01, 3, 5, 6, 50, and 60—illustrated with diagrams, to help you choose the right storage solution for speed and data protection.
This tutorial explains why and how to set up a software RAID1 on a CentOS 7 virtual machine using mdadm, covering prerequisites, disk partitioning, RAID creation, configuration, formatting, mounting, and common maintenance commands.
This article analyzes write latency caused by slab locking in older Linux kernels, evaluates kernel upgrades and BIOS power‑state settings, and explains CPU power‑management mechanisms (P‑States, C‑States, T‑States) that affect storage read/write performance.
This article provides a comprehensive introduction to modern distributed systems, covering recent research trends, practical technologies such as Paxos, Consistent Hashing, MapReduce, Spark, various storage and computing paradigms, and offers guidance for beginners on how to navigate the field.
This article explains the storage challenges of big data, introduces RAID technologies and their variants, and shows how the principles of RAID are applied in the Hadoop Distributed File System (HDFS) to achieve scalable, reliable, and high‑performance data storage and processing.
This article curates a selection of seminal industrial distributed system design papers—including Google’s GFS, MapReduce, Bigtable, Percolator, Spanner, F1, Amazon’s Dynamo and Aurora—providing brief insights and links for readers interested in foundational storage and database technologies.
This article presents SlimTrie, a trie‑based indexing structure that dramatically reduces memory consumption while maintaining fast query speeds, detailing its design, compression techniques, implementation steps, memory analysis, and performance comparisons with map and B‑Tree structures for large‑scale storage systems.
This article provides an in‑depth exploration of the NVMe over Fabrics protocol, detailing its design goals, data encapsulation scheme, IO transmission process, command extensions such as Connect and Property Get/Set, and the associated Linux reference implementation for RDMA and Fibre Channel transports.
The article explains the emergence of the NVMe over Fabrics protocol, its performance advantages for SSDs, the role of RDMA and other transport layers, and how shared, cloud‑native storage architectures are reshaping enterprise IT infrastructure.
This article details 360 Search's journey of constructing a Kubernetes‑based private cloud, covering its evolution, network and storage designs, logging pipeline, the Wayne management platform, encountered pitfalls, and future open‑source plans, offering practical insights for similar deployments.
This article explains the three main storage architectures—Direct Attached Storage (DAS), Network Attached Storage (NAS), and Storage Area Network (SAN)—and compares their protocols, scaling methods, caching policies, RAID levels, LUN concepts, and core Linux block I/O structures.
This guide explains how to use Linux's Logical Volume Manager (LVM) to create physical volumes, volume groups, and logical volumes, then demonstrates expanding and shrinking a logical volume safely, including filesystem resizing and removing a physical volume from a volume group.
This article explains the core differences between data lakes and data warehouses, their respective strengths, and how they complement each other to support both exploratory analytics and routine business reporting.
This article presents the integration of Alluxio as an in‑memory caching layer for QueryEngine's Spark SQL engine, detailing Alluxio's architecture, key features, deployment practice, performance testing methodology, results, and lessons learned for large‑scale ad‑hoc query acceleration.
The 2018 global storage merger‑and‑acquisition market shrank to just 36 deals—far below the 2006 peak of 104—highlighting a slowdown driven by industry maturity, scarce disruptive startups, and modest transaction values despite a history of multi‑billion‑dollar deals.
This guide explains essential Linux storage commands—including fdisk, df, du, mount, umount—covers automatic mounting via /etc/fstab, details disk partitioning with fdisk, and demonstrates how to format partitions using mkfs, providing practical examples for each operation.
Message queues serve as essential decoupling and traffic‑control components in distributed systems, and this article outlines their overall design—including producers, brokers, consumers, communication protocols such as JMS, AMQP and Kafka, storage options, consumption relationship handling, and advanced features like ordering, reliability, persistence, and transaction support.
The talk explains how Tencent’s serverless Mini Program Cloud Development platform eliminates the brain, cost, and coordination pains of backend creation by providing native, one‑click cloud functions, a MongoDB‑compatible database, and CDN‑accelerated storage, enabling single engineers to build scalable, reliable Mini Program services quickly and affordably.
This article introduces Zhuanzhuan's multi‑service storage layer—including MySQL, TiDB, ZZRedis, ZZKV/ZZList, and ZZOS—detailing their applicable scenarios, architectural designs, and the improvements made to address scalability, security, and resource‑usage challenges.
This article explains the evolution of storage networking, the performance bottlenecks of legacy SCSI‑based protocols, and how NVMe and NVMe over Fabrics leverage PCIe, RDMA, and high‑speed fabrics to deliver massive parallelism, lower latency, and a new paradigm for data‑center storage architectures.
NVMe SSD performance can be unpredictable, so this article opens the SSD "black box" to examine hardware, firmware, and workload factors—such as NAND type, multi‑queue design, garbage collection, and I/O patterns—and offers software‑level strategies to maximize flash efficiency in modern data‑center storage systems.
Microsoft's Volume Shadow Service (VSS) is a backup and recovery framework that creates consistent point-in-time snapshots by coordinating requestors, writers, and providers, supporting fast data backup, file-level restores, and various storage scenarios through full and copy-on-write snapshot methods.
This guide explains what a Network Attached Storage (NAS) is, why DIY or off‑the‑shelf NAS solutions may fall short for home users, and provides detailed hardware recommendations—including HP ProLiant MicroServer models—along with key criteria such as stability, expandability, energy efficiency, and ease of maintenance.
The article provides a comprehensive overview of the Network Data Management Protocol (NDMP), explaining its architecture, core concepts, backup mechanisms, version differences, supported backup software, and various network topologies such as 2‑way, 3‑way, and remote configurations for efficient NAS data protection.
This article provides a comprehensive overview of snapshot technology, explaining its creation process, various implementations across file systems, LVM, NAS, disk arrays, virtualization, and databases, and detailing mechanisms like COW, ROW, incremental and continuous protection while addressing consistency challenges.
This article provides a comprehensive introduction to Docker, covering its open‑source origins, client‑server architecture, networking and storage solutions, integration with PaaS/IaaS platforms, impact on DevOps practices, and its pivotal role in enabling modern microservice architectures.
This article explains how to build private and hybrid cloud solutions, outlines essential technology stacks, discusses scalability, resource management, and storage options, and provides practical guidance on improving availability and handling errors in cloud systems.
This guide explains LVM fundamentals and walks through creating physical volumes, volume groups, and logical volumes, formatting and mounting them, then demonstrates how to extend, shrink, and finally remove physical volumes safely using Linux command‑line tools.
This article explains the XFS file system’s key features such as data integrity, performance, scalability and bandwidth, and provides step‑by‑step instructions for downloading kernel patches, enabling XFS support, compiling the kernel, creating and mounting XFS partitions, and migrating existing Linux systems to XFS.
Storage snapshots are point‑in‑time, fully usable copies of data that protect against loss, enable testing and analysis, and are implemented mainly via copy‑on‑write or redirect‑on‑write techniques, each with distinct performance and deletion trade‑offs, so selecting the optimal method depends on workload and business requirements.
DataDirect Networks (DDN) announced a non‑binding intent to acquire all assets of Tintri, a storage vendor known for VM‑aware solutions, aiming to strengthen its portfolio with virtualization, AI real‑time analytics, and HPC capabilities, while the deal raises questions about market competition and the future positioning of both companies.
This article provides a comprehensive overview of the Lustre parallel file system, covering its architecture, key features, component roles, scalability, performance characteristics, and step‑by‑step configuration procedures for high‑performance computing environments.
This article explains Linux's hierarchical file system, detailing how the root directory, partitions, boot and super blocks, inodes, and data blocks interact to store and retrieve files, and it also covers file descriptors, sharing, and process forking.
This article explains how Weibo handles massive daily traffic of over a hundred billion requests by employing a five‑layer feed system and a six‑layer cache architecture that evolved from simple KV storage to sophisticated counter and existence‑check services, highlighting design choices, performance optimizations, and future directions.
The article provides a comprehensive overview of the Lustre file system, detailing its cluster storage architecture, core components, scalability, performance optimizations, security mechanisms, high‑availability features, and usage in high‑performance computing environments.
Shingled Magnetic Recording (SMR) uses overlapping tracks to dramatically increase disk surface density, lowering cost per gigabyte, but it eliminates random writes, requiring new zone‑based management and exposing standards like ZBC and ZAC for host‑aware and drive‑managed implementations.
This article demystifies reliability by clarifying the difference between durability and availability, exposing common misconceptions about MTBF, analyzing real‑world disk failure data, and presenting a practical formula for calculating the health probability of distributed storage systems.
This article reviews recent HPC market size and growth trends, highlights AI‑driven investment, and evaluates Huawei’s hardware, cloud, and industry‑specific solutions—including supercomputing, automotive, oil & gas, and entertainment—while offering personal insights on storage challenges and future directions.
The article examines the shifting landscape of high‑performance computing storage, detailing how Intel’s abandonment of Lustre, the rise of BeeGFS, and evolving demands from large‑scale analytics and AI are driving organizations to adopt more flexible, user‑space parallel file systems that promise better performance, scalability, and easier management.
This article surveys current storage research and industry developments, covering Open‑Channel SSD architecture, non‑volatile memory programming models, machine‑learning‑driven learned indexes, LSM‑Tree performance improvements, crash‑consistency verification, and recent virtualization and container‑storage advances.
This article provides a comprehensive overview of the NFSv4 protocol, its design goals, and the advantages of the user‑space NFS‑Ganesha implementation—including memory management, portability, simplified access, FUSE integration, modular architecture, threading, caching strategies, and its integration with Ceph RGW.
This article provides a comprehensive introduction to container technology—covering Docker's architecture, networking models, storage solutions, integration with PaaS/IaaS platforms, orchestration tools, and its impact on DevOps, micro‑services, and cloud computing ecosystems.
The article examines blockchain's future by highlighting three core technical constraints—high latency, costly computation, and limited storage—that must be overcome for the technology to move beyond speculation and achieve widespread practical use.
NVMe over Fabrics, leveraging PCIe and RDMA, replaces legacy SCSI and SAS protocols to deliver vastly higher bandwidth and lower latency for SSDs, enabling massive parallel I/O queues, and reshaping data‑center storage architectures through both FC‑NVMe and Ethernet‑based implementations.
This article details the architecture and practical techniques behind WeChat's large‑scale monitoring system, covering lightweight data collection, classification of real‑time, non‑real‑time and user‑specific metrics, anomaly detection algorithms, automated configuration, and high‑performance storage solutions for billions of events per minute.
This guide explains essential Linux storage commands—including fdisk, df, du, mount, and umount—covers automatic mounting via /etc/fstab, details disk partitioning with fdisk, and demonstrates how to format partitions using mkfs, providing practical examples for each step.
This article provides a detailed, curated list of the most popular Docker‑related tools across categories such as orchestration, CI/CD, monitoring, security, storage, networking and management, including brief descriptions, official links and cost information to help developers, DevOps engineers and platform architects choose the right solutions for every stage of the container lifecycle.
The 2017 KubeCon+CloudNativeCon in Austin showcased over 4,000 attendees, highlighted Kubernetes as the "Linux of the cloud," explored multi‑cloud strategies, service mesh adoption, storage challenges on Kubernetes, and underscored China's growing influence in the cloud‑native ecosystem.
The article provides a comprehensive technical analysis of NVMe over Fabric, highlighting the advantages of Fibre Channel over Ethernet, discussing RDMA, zero‑copy mechanisms, and the impact of SCSI translation layers on performance and adoption in modern data‑center storage architectures.
This guide explains why hard drives fail, distinguishes physical and logical damage, and reviews thirteen powerful data recovery utilities—complete with features, supported file systems, and usage tips—to help you restore lost files efficiently.
This article explains the ALUA (Asymmetric Logical Unit Access) multipathing model, compares it with A/A‑Symmetric and Active/Passive mechanisms, describes its application in dual‑active storage architectures, and outlines OS support and key features for improving I/O performance and reliability.
