Tagged articles
Erasure Coding
48 articles · Page 1 of 1
This comprehensive guide covers MinIO cluster deployment, erasure coding configuration, S3 API compatibility nuances, TLS/load balancing setup, user policies, monitoring, backup strategies, and troubleshooting common production issues like write quorum failures and signature mismatches.
Cluster OperationsErasure CodingMinIO
0 likes · 84 min read

RustFS 1.0 GA releases a production-ready, Apache 2.0-licensed S3-compatible object store written in Rust, achieving 2.3× MinIO performance on 4KB objects via SIMD-accelerated erasure coding, io_uring zero-copy I/O, and a GC-free runtime, while adding built-in Iceberg REST Catalog (S3 Tables) for data-lake workloads.
AI TrainingErasure CodingMinIO
0 likes · 18 min read

Baidu's Canghai storage evolves from siloed per-product systems to a unified architecture with MetaDB (distributed transactional metadata database) and Aries (scalable Blob KV data plane with online EC), using directory-aware sharding, conflict merging, Mantle path resolution, and hybrid EC/three-replica adaptation to support object, file, and block storage on one foundation.
AriesBlob KVErasure Coding
0 likes · 38 min read

RustFS, a Rust-written S3-compatible distributed object storage, hits GA with 32.5K GitHub stars, offering MinIO migration paths including on-demand data migration, versioning, erasure coding, and storage pool scaling, plus a Docker quick-start for production evaluation.
DockerErasure CodingMinIO
0 likes · 12 min read

This guide walks through the complete deployment of MinIO, an S3‑compatible object storage system, covering single‑node and erasure‑coded multi‑node clusters, hardware planning, TLS setup, bucket policies, lifecycle management, security hardening, monitoring with Prometheus, backup strategies, and detailed troubleshooting procedures, all backed by concrete commands and configuration examples.
Erasure CodingMinIOObject Storage
0 likes · 36 min read

This article explains how to integrate Ceph's erasure‑coded RBD pools with OpenStack, covering the design principles, storage pool layout, performance trade‑offs, and step‑by‑step configuration for Nova and Cinder to achieve higher storage utilization while maintaining high availability.
CephErasure CodingOpenStack
0 likes · 13 min read

This guide explains why traditional block and file storage struggle with massive unstructured data, introduces MinIO as a high‑performance, Go‑based S3‑compatible object storage, and provides step‑by‑step instructions for single‑node and erasure‑coded multi‑node deployments, TLS setup, client usage, policies, monitoring, backup, and troubleshooting.
Erasure CodingKubernetesMinIO
0 likes · 35 min read

This article explains how Erasure Coding (EC) improves data reliability and dramatically reduces storage overhead in Ozone by leveraging hot‑cold data characteristics, intelligent tiering, dynamic EC ratios, and repair throttling, while also discussing performance trade‑offs and limitations.
Data ReliabilityErasure CodingObject Storage
0 likes · 9 min read

This article details Zhihu's comprehensive cost‑reduction and efficiency‑boosting initiatives for its big‑data platform, covering FinOps‑driven financial operations, hybrid‑cloud architecture, cost allocation models, operational monitoring, and technical optimizations such as erasure coding, ZSTD compression, Spark auto‑tuning, and a remote shuffle service.
Cloud Cost ManagementCost OptimizationErasure Coding
0 likes · 22 min read

Baidu Canghai Storage’s unified technology stack—comprising a meta‑aware distributed metadata layer, a hybrid single‑node‑distributed namespace, and an online erasure‑coding data layer—delivers AI‑driven, high‑performance, low‑cost, ZB‑scale cloud storage by modularizing metadata, namespace, and data services for object, file, and block workloads.
BaiduErasure Codingcloud storage
0 likes · 16 min read

Baidu’s Canghai Storage unifies metadata, hierarchical namespace, and data layers into a Meta‑Aware, three‑generation architecture that scales to trillions of metadata items and zettabyte‑scale data, using a distributed transactional KV store, single‑machine‑distributed namespace, and online erasure‑coding micro‑services to deliver high performance, low cost, and seamless scalability.
Erasure CodingNewSQLbig data
0 likes · 18 min read

This article details Baidu’s unified storage architecture—covering its metadata, hierarchical namespace, and data layers—explaining how meta‑aware design, custom partitioning, flexible engines, and micro‑service based erasure coding together meet the scalability, performance, and cost demands of modern AI‑driven cloud storage workloads.
Erasure Codingcloud storagemetadata
0 likes · 17 min read

This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.
Erasure Codingappend enginebig data
0 likes · 14 min read

This article explains Baidu Intelligent Cloud's block storage (CDS) architecture, comparing fault‑tolerance methods, detailing the challenges of large‑scale erasure‑coded storage, and describing Baidu's two‑layer append‑engine solution that reduces I/O amplification while keeping costs low.
Erasure CodingI/O amplificationappend engine
0 likes · 15 min read

Combining academic advances and industry practice, the article surveys erasure‑coding techniques, then details Vivo’s optimized storage stack—enhancing Reed‑Solomon with bit‑matrix scheduling, parallel cross‑AZ repair, LRC and MSR layers, and intermediate‑result optimization—to achieve high reliability while minimizing bandwidth and storage overhead.
Erasure CodingRegenerating Codesdata redundancy
0 likes · 48 min read

Bilibili reduced petabyte‑scale storage costs by back‑porting erasure‑coding patches to its HDFS 2.8.4 cluster, deploying a parallel EC‑enabled cluster, adding a data‑proxy service, intelligent routing and block‑checking, and automating cold‑data migration, while noting write overhead and planning native acceleration.
Data ReliabilityErasure CodingHDFS
0 likes · 14 min read

This article explains how Zhihu adopted HDFS erasure coding to reduce storage costs, outlines cold‑hot file tiering policies, describes the EC conversion workflow and the custom EC Worker tool, and details methods for detecting and repairing damaged EC files in a Hadoop environment.
Data StorageErasure CodingHDFS
0 likes · 16 min read


ITPUBDec 25, 2022 · Cloud Native
This guide explains the fundamentals of distributed file systems, compares them with traditional storage, introduces MinIO’s architecture and features, and provides step‑by‑step instructions for deploying a multi‑node MinIO cluster with Nginx load balancing on Linux.
Erasure CodingLinuxMinIO
0 likes · 16 min read

This article explains how UCloud’s archival storage leverages inexpensive SMR disks, JBOD hardware, erasure‑coding blobs, and a power‑aware scheduling strategy to store massive cold data cheaply while still providing flexible, near‑real‑time retrieval when needed.
Erasure CodingJBODarchival storage
0 likes · 10 min read

This article presents a comprehensive overview of Baidu’s ARIES storage platform, detailing its design philosophy, architecture, key concepts, and engineering challenges, and explains how it underpins Baidu Netdisk’s massive data‑plane storage with high availability, cost‑performance trade‑offs, and robust monitoring.
Erasure CodingHigh Availabilitycloud storage
0 likes · 36 min read


ITPUBOct 20, 2022 · Big Data
The article examines why Hadoop's Distributed File System may become obsolete by detailing its three main shortcomings—deployment complexity, metadata memory limits, and high replication overhead—and explores how newer architectures and erasure coding could address these issues.
Erasure CodingHDFSbig data
0 likes · 8 min read

This article explains the shortcomings of HDFS at large scale, describes the Federation and Scaling approaches, and details how Apache Ozone redesigns metadata storage, introduces container abstraction, object semantics, and new features such as optimized OM, streaming writes, erasure coding, and RocksDB consolidation to improve scalability and performance.
Apache OzoneErasure CodingHDFS
0 likes · 11 min read

The article chronicles Bilibili’s 13‑day development of BOSS, a custom object storage service, detailing how it replaces MySQL‑based routing and ID generation with replicated etcd or Raft KV stores, models metadata via protobuf, adopts erasure coding and a Bitcask‑style engine, and implements safe delete, replica repair, and horizontal scaling for a resilient large‑scale system.
Erasure CodingObject Storagedistributed systems
0 likes · 28 min read

Kuaishou reduced half of its massive object storage expenses by redesigning its architecture to use HBase indexing, HDFS large‑file storage, MemoryCache, and a cross‑IDC LRC erasure‑coding warm layer that maintains disaster‑recovery while dynamically moving data from hot to warm to cold tiers.
Erasure CodingKuaishouLRC
0 likes · 12 min read

This article provides a comprehensive guide to optimizing Hadoop HDFS storage through erasure coding and heterogeneous storage policies, explains fault‑tolerance techniques such as safe mode and slow‑disk monitoring, and shares practical MapReduce performance tuning and enterprise‑level configuration examples for large‑scale clusters.
Erasure CodingHDFSHadoop
0 likes · 32 min read

This article introduces OPPO's self‑developed data lake storage system CBFS, covering the fundamentals of data lake storage, the multi‑layer CBFS architecture, its core technologies such as metadata management and erasure coding, and future directions for large‑scale, low‑cost data analytics.
CBFSErasure Codingbig data storage
0 likes · 14 min read

This article analyzes the quantitative model for data reliability in distributed storage, covering factors such as disk count, replication factor, recovery time, annualized failure rate, and copyset configuration, and derives formulas to estimate yearly data loss probability for both replica and erasure‑coding schemes.
AFRData ReliabilityErasure Coding
0 likes · 16 min read

This article details the end‑to‑end upgrade of a 5000‑node Hadoop 2.6.0 cluster to Hadoop 3.2.1 at 58.com, covering HDFS migration, RBF and EC adoption, Yarn federation and rolling upgrades, MR3 integration, extensive compatibility testing, and operational lessons learned for large‑scale big‑data platforms.
Erasure CodingHDFSHadoop
0 likes · 19 min read

This article details JD's end‑to‑end implementation of HDFS erasure coding, covering the migration from replication to EC, the three‑phase upgrade and rollback process, comprehensive automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and multi‑layer integrity safeguards to achieve significant storage cost reduction while maintaining reliability.
Data LifecycleErasure CodingHDFS
0 likes · 17 min read

This article details JD's practical experience migrating HDFS to erasure coding, covering the decision between upgrade and porting, the step‑by‑step upgrade and rollback procedures, automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and comprehensive data‑integrity safeguards to achieve significant storage cost reductions while maintaining production reliability.
Erasure CodingHDFScluster upgrade
0 likes · 17 min read

This article details a three‑part technical sharing that covers cost governance for offline Hadoop clusters, a large‑scale data‑center migration with architecture upgrades, and a tiered storage strategy using EC and COS to reduce storage costs and improve performance in a cloud‑native big‑data environment.
Big Data MigrationCOSErasure Coding
0 likes · 10 min read

This article provides a comprehensive overview of Red Hat Ceph’s distributed object‑storage architecture, covering storage pools, authentication, placement groups, the CRUSH algorithm, replication, erasure coding, internal operations, high‑availability mechanisms, client interfaces, and encryption, illustrated with diagrams and practical details.
CRUSHCephErasure Coding
0 likes · 40 min read

This article provides a comprehensive technical overview of Red Hat Ceph, covering its distributed object storage design, cluster architecture, storage pools, authentication, placement groups, CRUSH algorithm, I/O operations, replication, erasure coding, internal management tasks, high availability, client interfaces, data striping, and encryption mechanisms.
CRUSHCephData Striping
0 likes · 42 min read

Didi migrated HDFS to Hadoop 3.2 and implemented erasure coding—using XOR and Reed‑Solomon RS(6,3) striping—to replace three‑replica storage for cold data, building back‑ported clients, automated conversion tools, and cross‑datacenter backup pipelines, while addressing operational bugs and noting performance trade‑offs.
DidiErasure CodingHDFS
0 likes · 11 min read

This article provides a comprehensive technical overview of Red Hat Ceph, covering its distributed storage architecture, cluster components, storage pool types, authentication, placement algorithms, I/O paths, replication and erasure‑coding strategies, internal management operations, high‑availability mechanisms, client libraries, data striping, and encryption details.
CRUSHCephData Striping
0 likes · 39 min read

The team performed a rolling upgrade of HDFS from 2.7 to 3.2 on large clusters, resolving EditLog, Fsimage, StringTable and authentication incompatibilities by omitting EC data, using fallback images, rolling back commits and first upgrading to the latest 2.x release, following a staged JournalNode‑NameNode‑DataNode procedure, validating with rehearsals and a custom trash‑management tool, and achieving uninterrupted service, improved stability, performance and cost efficiency.
Cluster MigrationErasure CodingHDFS
0 likes · 11 min read

This article provides a detailed technical overview of EMC's Elastic Cloud Storage (ECS), covering its historical evolution, layered architecture, supported protocols, data protection mechanisms, performance characteristics, limitations, and future roadmap within the context of cloud object storage.
ECSElastic Cloud StorageErasure Coding
0 likes · 10 min read

This article provides a comprehensive technical overview of MinIO, an open‑source, S3‑compatible object storage system, covering its design philosophy, data organization, distributed architecture, erasure‑coding, lock management, lambda notifications, backup strategies, performance optimizations, and a comparative analysis with Ceph, highlighting its suitability for AI, big‑data, and cloud‑native deployments.
Erasure CodingMinIOObject Storage
0 likes · 22 min read


58 TechMay 21, 2019 · Backend Development
This article presents the architecture and key design features of WOS, the 58 Group’s self‑developed object storage system, covering its overall framework, proxy, store, directory, detector modules, fast‑upload “秒传” mechanism, and erasure‑coding solution for efficient, scalable, and reliable unstructured data storage.
Erasure CodingObject StorageWOS
0 likes · 12 min read

This article provides a comprehensive overview of Red Hat Ceph's distributed object storage architecture, covering storage pools, CRUSH placement, authentication, I/O workflows, internal operations, client interfaces, data striping, erasure coding, high availability, and encryption mechanisms for secure, scalable deployments.
CRUSHCephErasure Coding
0 likes · 40 min read

This article shares practical strategies used in a multi‑petabyte Hadoop environment to slash storage expenses, covering reduced replication, selective compression formats, tiered storage policies, and erasure coding, while weighing trade‑offs in reliability, performance, and operational complexity.
CompressionErasure CodingHDFS
0 likes · 10 min read

This article provides a comprehensive English translation of the Red Hat Ceph Storage Architecture Guide, covering Ceph's distributed object storage concepts, cluster architecture, storage pools, CRUSH algorithm, replication and erasure‑coding I/O, internal operations, high‑availability mechanisms, client interfaces, and encryption considerations for cloud environments.
CRUSHCephErasure Coding
0 likes · 40 min read

This article explains the challenges of growing data volumes, small files, and hot‑cold data in Hadoop HDFS, then details heterogeneous storage options, erasure‑coding techniques, and the open‑source SSM (Smart Storage Management) system that automates tiered storage based on data access patterns.
Erasure CodingHeterogeneous StorageSmart Storage Management
0 likes · 14 min read

The paper translates a USENIX'17 study on speculative partial writes for erasure‑coded storage, introducing PBS (Parity‑Based Speculation) which logs parity changes to turn costly random writes into sequential appends, achieving write IOPS and latency comparable to triple‑replication while maintaining fast recovery and high EC encoding throughput.
Erasure CodingPartial WritesSpeculative Writes
0 likes · 6 min read

The article details Tencent File System (TFS), describing its platform components, 3D indexing techniques, high‑performance key‑value storage (TSSD) with MHT, dual‑read and smooth scaling mechanisms, hybrid index storage, host‑level FTL, Append‑Only and erasure‑coding storage engines, and how these innovations deliver scalable, low‑cost, high‑performance data storage for massive workloads.
Erasure CodingIndexingKey-Value
0 likes · 12 min read

This talk explores how OpenStack Swift object storage can be optimized on SSDs, examines why object storage is gaining popularity, outlines operational concerns, discusses performance testing, SSD placement strategies, and erasure coding benefits, providing practical insights for cloud storage engineers.
Erasure CodingOpenStackSSD
0 likes · 29 min read

Facebook’s Cold Storage system redesigns hardware and software to store rarely accessed data with up to 75% lower power consumption, modular 2U racks holding 30 disks, Reed‑Solomon erasure coding for cheap redundancy, and a self‑healing “anti‑entropy” process that improves performance as the system scales.
Erasure CodingFacebookcold storage
0 likes · 12 min read

From the evolution of single‑machine file systems to today’s distributed, erasure‑coded cloud storage, this article examines why storage has become increasingly complex, the limitations of traditional replication, and how Qiniu’s next‑gen architecture leverages EC, faster repairs, and cost reductions to meet scalability, reliability, and availability demands.
Erasure Codingcloud storagereliability
0 likes · 15 min read
