This article explains how to integrate Ceph's erasure‑coded RBD pools with OpenStack, covering the design principles, storage pool layout, performance trade‑offs, and step‑by‑step configuration for Nova and Cinder to achieve higher storage utilization while maintaining high availability.
This guide explains why traditional block and file storage struggle with massive unstructured data, introduces MinIO as a high‑performance, Go‑based S3‑compatible object storage, and provides step‑by‑step instructions for single‑node and erasure‑coded multi‑node deployments, TLS setup, client usage, policies, monitoring, backup, and troubleshooting.
This article explains how Erasure Coding (EC) improves data reliability and dramatically reduces storage overhead in Ozone by leveraging hot‑cold data characteristics, intelligent tiering, dynamic EC ratios, and repair throttling, while also discussing performance trade‑offs and limitations.
This article details Zhihu's comprehensive cost‑reduction and efficiency‑boosting initiatives for its big‑data platform, covering FinOps‑driven financial operations, hybrid‑cloud architecture, cost allocation models, operational monitoring, and technical optimizations such as erasure coding, ZSTD compression, Spark auto‑tuning, and a remote shuffle service.
Baidu Canghai Storage’s unified technology stack—comprising a meta‑aware distributed metadata layer, a hybrid single‑node‑distributed namespace, and an online erasure‑coding data layer—delivers AI‑driven, high‑performance, low‑cost, ZB‑scale cloud storage by modularizing metadata, namespace, and data services for object, file, and block workloads.
Baidu’s Canghai Storage unifies metadata, hierarchical namespace, and data layers into a Meta‑Aware, three‑generation architecture that scales to trillions of metadata items and zettabyte‑scale data, using a distributed transactional KV store, single‑machine‑distributed namespace, and online erasure‑coding micro‑services to deliver high performance, low cost, and seamless scalability.
This article details Baidu’s unified storage architecture—covering its metadata, hierarchical namespace, and data layers—explaining how meta‑aware design, custom partitioning, flexible engines, and micro‑service based erasure coding together meet the scalability, performance, and cost demands of modern AI‑driven cloud storage workloads.
This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.
This article explains Baidu Intelligent Cloud's block storage (CDS) architecture, comparing fault‑tolerance methods, detailing the challenges of large‑scale erasure‑coded storage, and describing Baidu's two‑layer append‑engine solution that reduces I/O amplification while keeping costs low.
Combining academic advances and industry practice, the article surveys erasure‑coding techniques, then details Vivo’s optimized storage stack—enhancing Reed‑Solomon with bit‑matrix scheduling, parallel cross‑AZ repair, LRC and MSR layers, and intermediate‑result optimization—to achieve high reliability while minimizing bandwidth and storage overhead.
Bilibili reduced petabyte‑scale storage costs by back‑porting erasure‑coding patches to its HDFS 2.8.4 cluster, deploying a parallel EC‑enabled cluster, adding a data‑proxy service, intelligent routing and block‑checking, and automating cold‑data migration, while noting write overhead and planning native acceleration.
This article explains how Zhihu adopted HDFS erasure coding to reduce storage costs, outlines cold‑hot file tiering policies, describes the EC conversion workflow and the custom EC Worker tool, and details methods for detecting and repairing damaged EC files in a Hadoop environment.
This guide explains the fundamentals of distributed file systems, compares them with traditional storage, introduces MinIO’s architecture and features, and provides step‑by‑step instructions for deploying a multi‑node MinIO cluster with Nginx load balancing on Linux.
This article explains how UCloud’s archival storage leverages inexpensive SMR disks, JBOD hardware, erasure‑coding blobs, and a power‑aware scheduling strategy to store massive cold data cheaply while still providing flexible, near‑real‑time retrieval when needed.
This article presents a comprehensive overview of Baidu’s ARIES storage platform, detailing its design philosophy, architecture, key concepts, and engineering challenges, and explains how it underpins Baidu Netdisk’s massive data‑plane storage with high availability, cost‑performance trade‑offs, and robust monitoring.
The article examines why Hadoop's Distributed File System may become obsolete by detailing its three main shortcomings—deployment complexity, metadata memory limits, and high replication overhead—and explores how newer architectures and erasure coding could address these issues.
This article explains the shortcomings of HDFS at large scale, describes the Federation and Scaling approaches, and details how Apache Ozone redesigns metadata storage, introduces container abstraction, object semantics, and new features such as optimized OM, streaming writes, erasure coding, and RocksDB consolidation to improve scalability and performance.
The article chronicles Bilibili’s 13‑day development of BOSS, a custom object storage service, detailing how it replaces MySQL‑based routing and ID generation with replicated etcd or Raft KV stores, models metadata via protobuf, adopts erasure coding and a Bitcask‑style engine, and implements safe delete, replica repair, and horizontal scaling for a resilient large‑scale system.
Kuaishou reduced half of its massive object storage expenses by redesigning its architecture to use HBase indexing, HDFS large‑file storage, MemoryCache, and a cross‑IDC LRC erasure‑coding warm layer that maintains disaster‑recovery while dynamically moving data from hot to warm to cold tiers.
This article provides a comprehensive guide to optimizing Hadoop HDFS storage through erasure coding and heterogeneous storage policies, explains fault‑tolerance techniques such as safe mode and slow‑disk monitoring, and shares practical MapReduce performance tuning and enterprise‑level configuration examples for large‑scale clusters.
This article introduces OPPO's self‑developed data lake storage system CBFS, covering the fundamentals of data lake storage, the multi‑layer CBFS architecture, its core technologies such as metadata management and erasure coding, and future directions for large‑scale, low‑cost data analytics.
This article analyzes the quantitative model for data reliability in distributed storage, covering factors such as disk count, replication factor, recovery time, annualized failure rate, and copyset configuration, and derives formulas to estimate yearly data loss probability for both replica and erasure‑coding schemes.
This article details the end‑to‑end upgrade of a 5000‑node Hadoop 2.6.0 cluster to Hadoop 3.2.1 at 58.com, covering HDFS migration, RBF and EC adoption, Yarn federation and rolling upgrades, MR3 integration, extensive compatibility testing, and operational lessons learned for large‑scale big‑data platforms.
This article details JD's end‑to‑end implementation of HDFS erasure coding, covering the migration from replication to EC, the three‑phase upgrade and rollback process, comprehensive automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and multi‑layer integrity safeguards to achieve significant storage cost reduction while maintaining reliability.
This article details JD's practical experience migrating HDFS to erasure coding, covering the decision between upgrade and porting, the step‑by‑step upgrade and rollback procedures, automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and comprehensive data‑integrity safeguards to achieve significant storage cost reductions while maintaining production reliability.
This article details a three‑part technical sharing that covers cost governance for offline Hadoop clusters, a large‑scale data‑center migration with architecture upgrades, and a tiered storage strategy using EC and COS to reduce storage costs and improve performance in a cloud‑native big‑data environment.
This article provides a comprehensive overview of Red Hat Ceph’s distributed object‑storage architecture, covering storage pools, authentication, placement groups, the CRUSH algorithm, replication, erasure coding, internal operations, high‑availability mechanisms, client interfaces, and encryption, illustrated with diagrams and practical details.
This article provides a comprehensive technical overview of Red Hat Ceph, covering its distributed object storage design, cluster architecture, storage pools, authentication, placement groups, CRUSH algorithm, I/O operations, replication, erasure coding, internal management tasks, high availability, client interfaces, data striping, and encryption mechanisms.
Didi migrated HDFS to Hadoop 3.2 and implemented erasure coding—using XOR and Reed‑Solomon RS(6,3) striping—to replace three‑replica storage for cold data, building back‑ported clients, automated conversion tools, and cross‑datacenter backup pipelines, while addressing operational bugs and noting performance trade‑offs.
This article provides a comprehensive technical overview of Red Hat Ceph, covering its distributed storage architecture, cluster components, storage pool types, authentication, placement algorithms, I/O paths, replication and erasure‑coding strategies, internal management operations, high‑availability mechanisms, client libraries, data striping, and encryption details.
The team performed a rolling upgrade of HDFS from 2.7 to 3.2 on large clusters, resolving EditLog, Fsimage, StringTable and authentication incompatibilities by omitting EC data, using fallback images, rolling back commits and first upgrading to the latest 2.x release, following a staged JournalNode‑NameNode‑DataNode procedure, validating with rehearsals and a custom trash‑management tool, and achieving uninterrupted service, improved stability, performance and cost efficiency.
This article provides a detailed technical overview of EMC's Elastic Cloud Storage (ECS), covering its historical evolution, layered architecture, supported protocols, data protection mechanisms, performance characteristics, limitations, and future roadmap within the context of cloud object storage.
This article provides a comprehensive technical overview of MinIO, an open‑source, S3‑compatible object storage system, covering its design philosophy, data organization, distributed architecture, erasure‑coding, lock management, lambda notifications, backup strategies, performance optimizations, and a comparative analysis with Ceph, highlighting its suitability for AI, big‑data, and cloud‑native deployments.
This article presents the architecture and key design features of WOS, the 58 Group’s self‑developed object storage system, covering its overall framework, proxy, store, directory, detector modules, fast‑upload “秒传” mechanism, and erasure‑coding solution for efficient, scalable, and reliable unstructured data storage.
This article provides a comprehensive overview of Red Hat Ceph's distributed object storage architecture, covering storage pools, CRUSH placement, authentication, I/O workflows, internal operations, client interfaces, data striping, erasure coding, high availability, and encryption mechanisms for secure, scalable deployments.
This article shares practical strategies used in a multi‑petabyte Hadoop environment to slash storage expenses, covering reduced replication, selective compression formats, tiered storage policies, and erasure coding, while weighing trade‑offs in reliability, performance, and operational complexity.
This article provides a comprehensive English translation of the Red Hat Ceph Storage Architecture Guide, covering Ceph's distributed object storage concepts, cluster architecture, storage pools, CRUSH algorithm, replication and erasure‑coding I/O, internal operations, high‑availability mechanisms, client interfaces, and encryption considerations for cloud environments.
This article explains the challenges of growing data volumes, small files, and hot‑cold data in Hadoop HDFS, then details heterogeneous storage options, erasure‑coding techniques, and the open‑source SSM (Smart Storage Management) system that automates tiered storage based on data access patterns.
The paper translates a USENIX'17 study on speculative partial writes for erasure‑coded storage, introducing PBS (Parity‑Based Speculation) which logs parity changes to turn costly random writes into sequential appends, achieving write IOPS and latency comparable to triple‑replication while maintaining fast recovery and high EC encoding throughput.
The article details Tencent File System (TFS), describing its platform components, 3D indexing techniques, high‑performance key‑value storage (TSSD) with MHT, dual‑read and smooth scaling mechanisms, hybrid index storage, host‑level FTL, Append‑Only and erasure‑coding storage engines, and how these innovations deliver scalable, low‑cost, high‑performance data storage for massive workloads.
This talk explores how OpenStack Swift object storage can be optimized on SSDs, examines why object storage is gaining popularity, outlines operational concerns, discusses performance testing, SSD placement strategies, and erasure coding benefits, providing practical insights for cloud storage engineers.
Facebook’s Cold Storage system redesigns hardware and software to store rarely accessed data with up to 75% lower power consumption, modular 2U racks holding 30 disks, Reed‑Solomon erasure coding for cheap redundancy, and a self‑healing “anti‑entropy” process that improves performance as the system scales.
From the evolution of single‑machine file systems to today’s distributed, erasure‑coded cloud storage, this article examines why storage has become increasingly complex, the limitations of traditional replication, and how Qiniu’s next‑gen architecture leverages EC, faster repairs, and cost reductions to meet scalability, reliability, and availability demands.
