The article explains how Kuaishou partnered with Apache Hudi to overhaul its ODS‑based data lake, addressing latency, storage cost, and complexity for AI and BI workloads, detailing the evolution from mysql‑to‑hive to mysql‑to‑hudi 1.0 and 2.0, the resulting performance gains, cost savings, and future roadmap.
This article walks through Milvus 2.3‑2.6.x storage optimizations—Mmap, tiered storage, and clustering compaction—explaining their principles, configuration hierarchy, benchmark results, and concrete deployment templates that together can boost query performance up to 25‑fold while halving memory consumption.
This article dissects the end‑to‑end architecture of a billion‑scale short‑video platform, detailing layered design, core services such as upload, transcoding, recommendation, interaction, storage, and the key challenges of massive video storage, high‑concurrency streaming, low‑latency playback, and real‑time recommendation reliability.
This article explains how MySQL stores rows, compares the four InnoDB row formats, and evaluates the trade‑offs of using NULL versus explicit default values for nullable columns, covering storage overhead, indexing behavior, and best‑practice design recommendations.
This article explains how MySQL stores rows using different InnoDB row formats, illustrates the internal layout of variable‑length columns and hidden fields, and compares the trade‑offs of defining columns as NOT NULL versus allowing NULL values, offering practical guidance for database design.
This talk explores large‑scale AI training storage challenges and presents PCache, a high‑performance, cloud‑native caching system that optimizes metadata, read/write paths, deployment, and high‑availability, delivering significant throughput gains and cost savings for massive model training workloads.
Xiaomi’s engineers explain how they tackled data‑lake challenges—small files, metadata latency, and multi‑cloud costs—by combining compact storage, Gravitino‑based metadata governance, Iceberg and Paimon formats, and JuiceFS abstraction, achieving lower storage expenses, faster queries, and a roadmap toward intelligent, real‑time, multimodal lakehouses.
This article dissects the architecture of a modern short‑video backend, covering layered system design, core services such as video production, distribution, interaction, storage strategies, real‑time and offline recommendation engines, high‑concurrency streaming solutions, and practical techniques for cost control, scalability, and fault tolerance.
This comprehensive guide walks you through RAID fundamentals, hardware and software configuration, performance tuning, cost‑benefit analysis, fault diagnosis, and real‑world case studies, providing actionable commands and best‑practice recommendations to help you boost storage performance and reliability by up to 300%.
This article details Beike's data‑fusion architecture evolution, covering industry trends, multi‑stage Hadoop upgrades, storage cost optimization with erasure coding, remote shuffle integration, GPU‑centric training stability, and future hybrid‑cloud strategies, while also sharing organizational and operational lessons learned.
This article details the complex migration of a leading Southeast Asian tech group's data warehouse from Google BigQuery to Alibaba Cloud MaxCompute, outlining challenges such as storage format differences, SQL compatibility, and performance tuning, and explains how the new Append DeltaTable format with dynamic bucketing and incremental reclustering resolves these issues.
This article summarizes Alibaba Cloud’s key technical contributions at KubeCon China 2025, covering AI‑focused Kubernetes optimizations, Argo Workflows enhancements, storage strategies for large models, Fluid’s data orchestration, multi‑tenant security, and the RoleBasedGroup framework for PD‑separated AI inference.
This article explains how Erasure Coding (EC) improves data reliability and dramatically reduces storage overhead in Ozone by leveraging hot‑cold data characteristics, intelligent tiering, dynamic EC ratios, and repair throttling, while also discussing performance trade‑offs and limitations.
This article details how Kuaishou dramatically cut storage costs and improved cache efficiency for its core Photo data object by cleaning up redundant JSON fields, applying selective serialization, and performing large‑scale data cleaning, achieving a 25% size reduction, a 2% cache‑hit increase, and multi‑hundred‑TB savings.
This article explains the principles, configuration steps, monitoring metrics, leader selection, data migration granularity, compaction, invalid data cleanup, and cache mechanisms of cold‑hot data separation in Baidu Intelligent Cloud's Palo for Apache Doris, illustrating how tiered storage reduces costs while maintaining query performance.
This article examines the growing challenges of data management—such as asset discoverability, architectural rigidity, development quality, and rising resource costs—and presents a comprehensive data‑governance framework that leverages standards, agile architecture, development isolation, and active‑metadata‑driven lifecycle evaluation to improve efficiency, reduce expenses, and enable intelligent, automated data back‑filling.
The article outlines JD Retail's data management challenges—including asset awareness, architectural agility, development quality, and rising resource costs—and presents a comprehensive data governance framework that leverages data standards, agile architecture, development isolation, resource optimization, and active metadata to achieve intelligent lifecycle evaluation, automated back‑fill, and future‑oriented data fabric improvements.
This article explores practical performance testing and tuning techniques, focusing on storage model optimization and call‑chain analysis to improve hot‑SKU inventory pre‑occupancy throughput, presenting detailed pressure‑testing scenarios, results, cache‑layer redesign, and strategies for identifying and mitigating system bottlenecks.
This article outlines a comprehensive, data‑driven methodology for reducing MySQL storage expenses—including background analysis, challenge identification, a nine‑grid systematic framework, benefit estimation, safety and stability verification, gray‑scale rollout, and rollback strategies—demonstrating over 50% disk space savings in a large‑scale billing system.
Apache Pulsar, a cloud‑native distributed messaging platform, solves Didi Big Data’s DKafka bottlenecks by separating compute and storage, using sequential log writes, heterogeneous disks, multi‑level caching, bundle‑based load balancing and automatic scaling, dramatically improving stability while introducing richer monitoring complexity.
This article presents a systematic, nine‑grid method for reducing MySQL storage expenses—including table compression, JSON field serialization, and hot‑cold data separation—while quantifying benefits, ensuring data safety, and validating system stability through staged testing and SRE metrics.
This article outlines the background, challenges, systematic methods, benefit calculations, data‑safety and stability checks, verification steps, rollback strategies, and gray‑deployment practices for lowering MySQL storage expenses in large‑scale billing systems while maintaining system reliability.
The article explains how a meeting control system tackled severe storage pressure from high concurrent traffic by introducing a proxy layer, multi‑active disaster‑recovery, identity‑based data isolation, dynamic‑static key separation, multi‑level caching, overload protection, sharding with dual‑write migration, and extensive monitoring to meet 100k QPS and ensure reliability.
The article explains why a Linux server can run out of inodes despite having plenty of free disk space, demonstrates how creating an ext4 filesystem with the -T small option dramatically increases inode count, and discusses the performance trade‑offs and quick fixes for inode exhaustion.
The Venus log platform at iQiyi migrated from an ElasticSearch‑Kibana architecture to an Iceberg‑based data lake with Trino, cutting storage and compute costs by over 70%, boosting stability by 85%, and efficiently supporting billions of daily logs through write‑heavy, low‑query workloads.
This article explains Baidu Intelligent Cloud's block storage (CDS) architecture, comparing fault‑tolerance methods, detailing the challenges of large‑scale erasure‑coded storage, and describing Baidu's two‑layer append‑engine solution that reduces I/O amplification while keeping costs low.
Didi’s redesigned Spatial‑Temporal Supply‑Demand System replaces a single‑Redis bottleneck with a multi‑cluster routing layer, semantic sharding, multi‑level caching and delayed queues, achieving higher horizontal scalability, fault isolation, ~30 % latency reduction, increased cache hit rates, fewer query nodes, and faster, code‑free feature configuration.
This article explains Alibaba Cloud's data lake architecture, unified metadata services, storage management optimizations, and format handling techniques, illustrating how lakehouse concepts, multi‑engine support, and lifecycle policies enable efficient, secure, and cost‑effective big data processing in the cloud.
This article analyzes the EROFS cluster modes (INFLIGHT, HOOKED, FOLLOWED, FOLLOWED_NOINPLACE) in Linux kernel 6.x, explaining how they determine whether in-place I/O can be used based on the current status of pclusters in the chain.
This article details Bilibili's evolution of big data governance, describing the early data growth challenges, the launch of the "Wanglou" project, the development of asset metadata and governance indicator frameworks, storage cost reduction strategies, scoring models, and the shift from passive, single‑point fixes to proactive, multi‑dimensional governance across the organization.
Bilibili’s exabyte‑scale big‑data platform, after rapid growth created fragmented ownership and costly storage, launched the Wanglou project to build a metadata‑driven, indicator‑based governance framework that cut storage use by half, introduced compliance scoring and automation, and now plans to extend proactive, multi‑dimensional governance to compute, traffic and lake‑house resources.
The article describes how an e‑commerce product ranking system was migrated to the new “Liao Yue” platform, decoupling it from the search module, introducing fresh metrics and Elasticsearch‑based sorting, then optimizing storage by separating B‑end and C‑end data—cutting costs 60%—with a gray‑scale rollout, dual‑read validation, rollback safeguards, and completing the two‑week, zero‑failure migration that delivered a closed‑loop, faster iteration system.
This article analyzes the shift from traditional multi‑layer data warehouse dimensional modeling to a single-layer wide‑table approach, detailing business drivers, technical architecture, storage and query performance gains, as well as the development, maintenance, and operational challenges involved.
The article explains JPEG’s lossy fundamentals, introduces Lepton’s lossless layer and its optimizations—such as arithmetic coding and multithreaded Huffman switching—and describes how vivo’s hybrid physical‑server and Kubernetes deployment achieves roughly 22 % storage reduction across petabytes of JPEG images despite high CPU demands.
The article proposes replacing the traditional multi‑layered data‑warehouse architecture (ODS‑DWD‑DWS‑ADS) with a single, column‑store wide‑table per business theme, achieving roughly 30 % storage savings and faster queries, while acknowledging higher ETL complexity, back‑tracking costs, and production timing challenges.
This article explains how to design a high‑performance, low‑cost log retrieval architecture that avoids Elasticsearch by partitioning logs into time‑based chunks, indexing only metadata, using multi‑tier storage (local, remote, archive), and orchestrating queries through GD‑Search, Local‑Search, Remote‑Search and Log‑Manager components.
ByteDance’s Data Catalog article details the system’s unified metadata model, standardized ingestion connectors, search optimization techniques, lineage capabilities, and storage layer enhancements, highlighting key technical designs, performance improvements, and future work to advance data governance and asset utilization.
The article outlines DeWu’s warehouse‑management recommendation system, which combines an online‑near‑line‑offline architecture to quickly recall viable shelf slots and rank them by space utilization, travel time, and sales potential, enabling automated, constraint‑aware placement that cuts picking time and inventory costs.
Bilibili’s HDFS now runs a three‑tier architecture—access, metadata, and data layers—enhanced with a custom MergeFS router, observer NameNode, dynamic load balancing, fast‑failover pipelines, and storage‑aware policies, while future work targets transparent erasure coding, tiered data routing, lock refinements, and a Hadoop 3.x migration.
The article analyzes the memory overhead of naïvely storing per‑user read/unread lists for group chat messages and proposes a bitmap‑based scheme with user‑to‑map ID mapping, quit flags, and compact storage that can reduce per‑message space by over 95 percent.
This article details YiBei's advertising data platform built on ClickHouse, covering business requirements, why ClickHouse was chosen over Druid, storage engine and compression choices, real‑time and offline ingestion pipelines, partitioning, Zookeeper bottlenecks, atomic data replacement, and testing and release strategies for a high‑throughput, low‑latency ad analytics system.
This article examines how to model read/unread status for group chat messages using bitmap techniques, evaluates memory costs of naive approaches, and presents a compact storage schema that handles member joins, exits, and scalability while drastically reducing per‑message overhead.
This article explains the fundamentals of message queues, describing how MQ, Kafka, and RocketMQ handle asynchronous processing, traffic shaping, and decoupling through topics, partitions, consumer groups, and storage mechanisms such as sequential writes and page‑cache optimization.
This article presents a comprehensive overview of using Apache Iceberg with object storage to construct scalable data lake solutions, covering lake architecture, Iceberg table organization, Flink‑based write and read workflows, catalog abstractions, object storage versus HDFS comparisons, append‑upload and atomic‑commit challenges, a demonstration setup, and ideas for storage optimization.
This article details JD's end‑to‑end implementation of HDFS erasure coding, covering the migration from replication to EC, the three‑phase upgrade and rollback process, comprehensive automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and multi‑layer integrity safeguards to achieve significant storage cost reduction while maintaining reliability.
This article details JD's practical experience migrating HDFS to erasure coding, covering the decision between upgrade and porting, the step‑by‑step upgrade and rollback procedures, automated testing, a custom data‑lifecycle management system for hot‑warm‑cold data, and comprehensive data‑integrity safeguards to achieve significant storage cost reductions while maintaining production reliability.
Elasticsearch 7.10 adds searchable snapshots, letting users query indices stored directly in remote repositories such as S3 or COS, which halves storage costs, decouples storage from compute, supports manual mounting and ILM cold‑phase policies, and promises future full storage‑compute separation without local caching.
This article introduces Redis's advanced bitmap capabilities, detailing the SETBIT, GETBIT, BITCOUNT, and BITOP commands, their syntax, underlying SDS data structure, performance characteristics, storage calculations, and practical use cases such as user sign‑in tracking and online status monitoring.
Over seven months, Didi’s Elasticsearch team upgraded more than 30 clusters, 2,000 nodes and 4 PB of data from version 2.3.3 to 6.6.1, overcoming protocol and mapping incompatibilities with a multi‑version Arius Gateway, custom Java SDK, ECM and AMS, while saving 1 PB of storage, decommissioning 400 machines, boosting query speed by 40 %, write throughput by 30 % and cutting CPU use 10 % for an estimated 80 w/month cost reduction.
This article examines how to select and configure disk solutions—single‑disk, multi‑directory, RAID, and LVM—for Apache Kafka deployments, comparing performance, cost, scalability, and reliability to help operators build stable, high‑throughput messaging infrastructures.
The article explains how Alluxio, a memory‑speed virtual distributed file system, acts as a virtual data lake to unify access to structured and unstructured big‑data across heterogeneous storage systems, offering on‑demand fast local access, intelligent caching, reduced storage costs, and enterprise‑grade security and fault tolerance.
The article analyzes the evolution of data protection—from manual copies and scripts to snapshots, Continuous Data Protection (CDP), and Copy Data Management (CDM)—highlighting their technical mechanisms, benefits such as near‑zero RPO, implementation models, vendor landscape, and key considerations for selecting the right solution in modern cloud‑centric environments.
This article shares practical strategies used in a multi‑petabyte Hadoop environment to slash storage expenses, covering reduced replication, selective compression formats, tiered storage policies, and erasure coding, while weighing trade‑offs in reliability, performance, and operational complexity.
This article introduces OpenTSDB as a distributed, scalable time‑series database built on HBase, explains its architecture, data model, and storage optimizations, presents real‑world monitoring use cases, analyzes performance issues caused by high‑cardinality tags, and details the solution steps taken to restore query speed.
iQIYI built a cloud‑based deep‑learning training platform called Jarvis, replacing the initial Runonce service, by containerizing GPU tasks, adopting Ceph S3 storage with FUSE, optimizing data pipelines, and addressing compute, storage, and networking challenges to improve scalability and reduce GPU idle time.
The article explains how Alluxio provides a virtual distributed file system that acts as a "virtual data lake," enabling unified, high‑performance access to structured and unstructured data across heterogeneous storage back‑ends while reducing storage costs through intelligent caching and eliminating the need for permanent data copies.
This article shares a detailed, experience‑driven guide on Elasticsearch tuning, covering data model fundamentals, storage cost reductions, cluster stability tricks, performance‑boosting settings, and custom kernel improvements, all illustrated with real‑world diagrams and Q&A insights.
This article explains the architecture of Hadoop HDFS, identifies performance bottlenecks in page cache and metadata handling on DataNodes, and presents four practical optimization techniques—including cache‑buffer separation, barrier disabling, directory restructuring, and real‑time monitoring—demonstrating significant throughput and latency improvements in large‑scale clusters.
This article explains how to assess I/O capabilities, choose appropriate evaluation tools, monitor key performance metrics, and apply targeted optimization techniques for both disk and network I/O to improve system throughput and latency.
This article provides a comprehensive guide to optimizing Lustre, the leading open‑source parallel file system for high‑performance computing, covering network bandwidth, stripe settings, client configuration, RAID choices, small‑file handling, and practical system commands to improve aggregate I/O performance.
The article presents a comprehensive solution for reducing Hadoop cluster storage consumption by applying ZFS‑based transparent compression and data‑governance techniques, detailing the technical background, design choices, implementation steps, performance optimizations, and observed storage savings.
The article explains how Ctrip's big‑data platform introduced Alluxio to isolate real‑time Spark Streaming jobs from HDFS NameNode maintenance, reduce NameNode pressure, improve Spark SQL performance, and provide a unified storage layer across multiple HDFS clusters.
This guide explains why Linux I/O scheduler selection matters for virtualized servers, compares deadline, CFQ, noop and anticipatory schedulers, and shows how to configure them globally or per‑disk to improve storage performance in modern data‑center environments.
The article explains the background of flash‑memory wear‑leveling, reviews common garbage‑collection strategies, compares classic algorithms such as Greedy, Cost‑Benefit, CAT and CICL, and introduces the Non‑Balance method that evaluates real block endurance to extend SSD life up to three times.
This article examines Alibaba's strategies for designing and deploying custom x86 processors, SSDs, high‑performance storage solutions, and next‑generation chassis to meet the massive, diverse demands of its global data‑center infrastructure while optimizing cost, energy efficiency, and scalability.
During high‑concurrency flash‑sale events, Oracle’s log file sync became a performance bottleneck; the article analyzes storage, OS, and Oracle Disk Manager factors, presents AWR metrics, demonstrates tuning steps—including disabling adaptive log file sync and enabling ODM—and shows measurable latency reductions.
The talk by Twitter engineer Henna Kermani outlines how the Media Platform’s decoupled, resumable upload pipeline, handle‑based storage, TTL policies, on‑demand processing, and Progressive JPEG adoption enabled processing 3,000 images per second while cutting storage, compute costs and improving operational flexibility.
This article examines why storage capacity limits IT systems, explains how database compression reduces disk usage and I/O time, discusses various dictionary‑based compression methods, related operations and commands, and evaluates compression ratios and overall impact on system performance.
This article investigates how long‑term use of Android devices leads to storage fragmentation and write amplification, explains the underlying NAND flash mechanisms, and evaluates TRIM‑based solutions such as discard and fstrim for restoring I/O performance.
This article explains data deduplication technology, its definition, various classification schemes based on execution time, block size, granularity, and location, and compares it with data compression, highlighting how both techniques can be combined to maximize storage savings.
Facebook’s data warehouse, storing over 300 PB and growing by 600 TB daily, transitioned from the RCFile format to an optimized ORCFile implementation, achieving 5‑8× better compression and up to three‑fold faster write performance while maintaining high read efficiency.
The article examines the inevitability of service coupling in large‑scale web applications and proposes a two‑dimensional architecture that separates business and logic layers, uses internal data stores, introduces a naming‑and‑location service, selects appropriate transport and RPC protocols, and automates operations with health checks, load balancing, and failover to achieve continuous reliability.
