0 views collected around this technical thread.
This article explains how Erasure Coding (EC) improves data reliability and dramatically reduces storage overhead in Ozone by leveraging hot‑cold data characteristics, intelligent tiering, dynamic EC ratios, and repair throttling, while also discussing performance trade‑offs and limitations.
This article details how Kuaishou dramatically cut storage costs and improved cache efficiency for its core Photo data object by cleaning up redundant JSON fields, applying selective serialization, and performing large‑scale data cleaning, achieving a 25% size reduction, a 2% cache‑hit increase, and multi‑hundred‑TB savings.
The article outlines JD Retail's data management challenges—including asset awareness, architectural agility, development quality, and rising resource costs—and presents a comprehensive data governance framework that leverages data standards, agile architecture, development isolation, resource optimization, and active metadata to achieve intelligent lifecycle evaluation, automated back‑fill, and future‑oriented data fabric improvements.
This article explores practical performance testing and tuning techniques, focusing on storage model optimization and call‑chain analysis to improve hot‑SKU inventory pre‑occupancy throughput, presenting detailed pressure‑testing scenarios, results, cache‑layer redesign, and strategies for identifying and mitigating system bottlenecks.
NetEase Cloud Music’s RTA advertising system delivers real‑time, personalized ads at massive scale by using isolated Nginx clusters, layered decoupling, asynchronous Netty/Redis processing, and optimized storage with hash‑based key compression and Protostuff serialization, while supporting automated audience selection and in‑app attribution to boost user acquisition.
Apache Pulsar, a cloud‑native distributed messaging platform, solves Didi Big Data’s DKafka bottlenecks by separating compute and storage, using sequential log writes, heterogeneous disks, multi‑level caching, bundle‑based load balancing and automatic scaling, dramatically improving stability while introducing richer monitoring complexity.
This article outlines the background, challenges, systematic methods, benefit calculations, data‑safety and stability checks, verification steps, rollback strategies, and gray‑deployment practices for lowering MySQL storage expenses in large‑scale billing systems while maintaining system reliability.
The article explains how a meeting control system tackled severe storage pressure from high concurrent traffic by introducing a proxy layer, multi‑active disaster‑recovery, identity‑based data isolation, dynamic‑static key separation, multi‑level caching, overload protection, sharding with dual‑write migration, and extensive monitoring to meet 100k QPS and ensure reliability.
The Venus log platform at iQiyi migrated from an ElasticSearch‑Kibana architecture to an Iceberg‑based data lake with Trino, cutting storage and compute costs by over 70%, boosting stability by 85%, and efficiently supporting billions of daily logs through write‑heavy, low‑query workloads.
Didi’s redesigned Spatial‑Temporal Supply‑Demand System replaces a single‑Redis bottleneck with a multi‑cluster routing layer, semantic sharding, multi‑level caching and delayed queues, achieving higher horizontal scalability, fault isolation, ~30 % latency reduction, increased cache hit rates, fewer query nodes, and faster, code‑free feature configuration.
This article explains Alibaba Cloud's data lake architecture, unified metadata services, storage management optimizations, and format handling techniques, illustrating how lakehouse concepts, multi‑engine support, and lifecycle policies enable efficient, secure, and cost‑effective big data processing in the cloud.
This article analyzes the EROFS cluster modes (INFLIGHT, HOOKED, FOLLOWED, FOLLOWED_NOINPLACE) in Linux kernel 6.x, explaining how they determine whether in-place I/O can be used based on the current status of pclusters in the chain.
Bilibili’s exabyte‑scale big‑data platform, after rapid growth created fragmented ownership and costly storage, launched the Wanglou project to build a metadata‑driven, indicator‑based governance framework that cut storage use by half, introduced compliance scoring and automation, and now plans to extend proactive, multi‑dimensional governance to compute, traffic and lake‑house resources.
Bilibili reduced petabyte‑scale storage costs by back‑porting erasure‑coding patches to its HDFS 2.8.4 cluster, deploying a parallel EC‑enabled cluster, adding a data‑proxy service, intelligent routing and block‑checking, and automating cold‑data migration, while noting write overhead and planning native acceleration.
The article describes how an e‑commerce product ranking system was migrated to the new “Liao Yue” platform, decoupling it from the search module, introducing fresh metrics and Elasticsearch‑based sorting, then optimizing storage by separating B‑end and C‑end data—cutting costs 60%—with a gray‑scale rollout, dual‑read validation, rollback safeguards, and completing the two‑week, zero‑failure migration that delivered a closed‑loop, faster iteration system.
This article analyzes the shift from traditional multi‑layer data warehouse dimensional modeling to a single-layer wide‑table approach, detailing business drivers, technical architecture, storage and query performance gains, as well as the development, maintenance, and operational challenges involved.
The article explains JPEG’s lossy fundamentals, introduces Lepton’s lossless layer and its optimizations—such as arithmetic coding and multithreaded Huffman switching—and describes how vivo’s hybrid physical‑server and Kubernetes deployment achieves roughly 22 % storage reduction across petabytes of JPEG images despite high CPU demands.
The article proposes replacing the traditional multi‑layered data‑warehouse architecture (ODS‑DWD‑DWS‑ADS) with a single, column‑store wide‑table per business theme, achieving roughly 30 % storage savings and faster queries, while acknowledging higher ETL complexity, back‑tracking costs, and production timing challenges.
ByteDance’s Data Catalog article details the system’s unified metadata model, standardized ingestion connectors, search optimization techniques, lineage capabilities, and storage layer enhancements, highlighting key technical designs, performance improvements, and future work to advance data governance and asset utilization.
The article outlines DeWu’s warehouse‑management recommendation system, which combines an online‑near‑line‑offline architecture to quickly recall viable shelf slots and rank them by space utilization, travel time, and sales potential, enabling automated, constraint‑aware placement that cuts picking time and inventory costs.