This article explains how Erasure Coding (EC) improves data reliability and dramatically reduces storage overhead in Ozone by leveraging hot‑cold data characteristics, intelligent tiering, dynamic EC ratios, and repair throttling, while also discussing performance trade‑offs and limitations.
This article introduces a newly released comprehensive guide on SSD flash technology while providing an in‑depth analysis of emerging Storage Class Memory (SCM) technologies—such as PCM, ReRAM, MRAM, and NRAM—detailing their principles, current research challenges, and potential system‑level innovations.
Didi’s multi‑year data‑stability program for its customer‑service platform progressed through fault‑centered engineering, business‑aligned cross‑team work, and capability normalization, instituting pre‑, mid‑ and post‑fault safeguards, clear ownership, automated alerts and repair tools, which cut fault count by 42 % and more than doubled mean‑time‑to‑repair while boosting team communication and satisfaction.
This article explains how Apache Pulsar combines messaging, storage, and lightweight function computing into a cloud‑native streaming platform, detailing its architecture, storage‑compute separation, tiered storage, pluggable protocols, reliability guarantees, and rich ecosystem compared with traditional queues and Kafka.
MySQL replication copies data changes from a primary server to one or more replicas using binlog events—supporting statement, row, or mixed formats and GTID positioning—to provide real‑time backup, read‑write separation, high‑availability failover, and integration pipelines via asynchronous, semi‑synchronous, or centralized binlog services.
Bilibili reduced petabyte‑scale storage costs by back‑porting erasure‑coding patches to its HDFS 2.8.4 cluster, deploying a parallel EC‑enabled cluster, adding a data‑proxy service, intelligent routing and block‑checking, and automating cold‑data migration, while noting write overhead and planning native acceleration.
This article explains the design, implementation, and future plans of Baidu's log middle platform, detailing its lifecycle management, service architecture, data reliability goals of eliminating duplication and loss, and the technical measures taken across SDKs, servers, and streaming pipelines to achieve near‑100% data integrity.
This article describes Baidu's log middle platform architecture, its data lifecycle management, integration status, terminology, service overview, core challenges of ensuring data accuracy, and the implemented optimizations for persistent storage, service decomposition, and SDK reporting to achieve near‑100% no‑repeat no‑loss reliability.
This article introduces the current state of Baidu's log platform, explains its lifecycle from data collection to downstream applications, analyzes the challenges of achieving near‑zero duplication and loss, and presents architectural optimizations and best‑practice recommendations to improve data stability and accuracy across the system.
This article explains Kafka's message delivery semantics, identifies the exact scenarios where data can be lost in producer, broker, and consumer components, and provides concrete configuration and coding practices to ensure reliable, at‑least‑once or exactly‑once delivery in production environments.
This article analyzes the quantitative model for data reliability in distributed storage, covering factors such as disk count, replication factor, recovery time, annualized failure rate, and copyset configuration, and derives formulas to estimate yearly data loss probability for both replica and erasure‑coding schemes.
UCloud’s US3 archive storage leverages high‑density SMR drives combined with JBOD architecture and a custom IO‑scheduling algorithm to slash hardware CAPEX by up to 80%, reduce OPEX electricity costs, and maintain data reliability through embedded metadata and dual‑head redundancy.
This article details how 360 adopted Cassandra for its cloud drive, describing Cassandra’s decentralized architecture, the reasons for its selection over HBase, large‑scale deployment challenges, performance optimizations, reliability improvements, disk utilization techniques, and the evolution of the system from 2010 to present.
The article explains how OceanBase, a distributed enterprise‑grade database, achieves strong data reliability and rapid service recovery on ordinary PC servers by combining Paxos‑based consensus, enhanced redo‑log verification, periodic checkpoint checks, and fine‑grained fail‑over mechanisms, surpassing traditional hardware‑dependent databases.
This article presents a comprehensive data quality model derived from ISO 9126, maps its characteristics to data testing, outlines practical testing methods and tool requirements, and demonstrates how to integrate quality checks into the data development lifecycle for reliable, efficient big‑data pipelines.
This article explains the core principles of message queues, outlines common data‑loss scenarios for RabbitMQ and Kafka, and provides practical techniques—such as transactions, confirm mode, persistence settings, and replication configurations—to ensure reliable, loss‑free messaging.
This article explains Kafka's mechanisms for ensuring data reliability through partition replicas, producer acknowledgments, and leader election, and describes how the high‑water‑mark and ISR concepts maintain strong data consistency across the cluster.
At the 2023 Tencent Cloud + Future summit, Liu Jinming detailed Tencent Cloud COS’s three‑tier EB‑level object storage architecture—covering network, application, and data layers—highlighting its 99.95% availability, 11‑nine durability, end‑to‑end encryption, scalable performance, tiered pricing, and real‑world media and security use cases.
The article examines the recurring bugs and reliability issues in Linux file systems—especially ext4—highlighting Linus Torvalds' criticism, POSIX shortcomings, fuzz‑testing results, historical bug statistics, and practical advice for developers to avoid data‑loss pitfalls.
Facing a 0.02% SQLite corruption rate that threatened years of chat history, the WeChat mobile team identified three main causes—insufficient space, power loss, and sync failures—and implemented space management, full sync settings, and master‑table backups, halving damage and doubling repair success.
