The article details JD’s data lake architecture, its 500 PB scale, self‑developed Hudi extensions—including LSM‑Tree‑based MoR tables, custom indexing, IO optimizations, Flink stream scheduling, and NativeIO SDK—along with benchmarks, community contributions, and future roadmap for real‑time big‑data processing.
This article explains how to improve Paimon's query performance by optimizing table schemas, storage settings, query parameters, and index designs, covering table mode choices, partitioning, file formats, parallelism, batch reads, and various index types such as Bloom filters and clustering indexes.
This article walks you through the inner workings of modern databases—explaining storage engines, query parsing, execution, and transaction models—while comparing classic B‑Tree structures with newer LSM‑Tree designs to help you decide whether SQL or NoSQL best fits your performance and consistency needs.
The article walks through a complete Zig implementation of an LSM‑Tree, detailing its in‑memory skip‑list MemTable, immutable SSTable blocks with compression and Bloom filters, write‑ahead logging, iterator hierarchy for reads, and multi‑level compaction logic that merges and rewrites SSTables.
This article provides a comprehensive overview of Apache Paimon, covering its real‑time lake ingestion, unified stream‑batch processing, table types (primary‑key and append‑only), LSM‑tree storage, bucket mechanisms, merge‑engine options, compaction strategies, concurrency control, consumption methods, tag management, data cleanup, and system tables for big‑data workloads.
This article walks through building a database storage engine from a simple shell script to a full distributed key‑value system, covering in‑memory indexing, SSTable creation, LSM‑Tree architecture with compaction, replication strategies, and sharding techniques for scaling across multiple machines.
The article explains Apache Hudi’s three‑layer architecture and details four major 1.0 enhancements—LSM‑tree timeline, non‑blocking concurrency control, file‑group reader/writer APIs, and function indexes—while providing a brief review and links to the Hudi 1.x RFC.
Modern databases have largely shifted from B‑tree based storage to LSM‑tree engines due to SSD hardware characteristics, high‑write workloads, concurrency advantages, simpler implementation, and evolving application demands, with additional insights into Paxos/Raft consensus, common database jargon, and performance optimizations.
This article explains Paimon's core concepts—including snapshots, partitions, buckets, consistency guarantees, file layout, LSM‑tree organization, and compaction strategies—while also covering table management tasks such as snapshot expiration, rollback, partition expiration, and small‑file mitigation techniques.
The article explains the Log-Structured Merge-Tree (LSM) architecture, its core components, advantages and disadvantages, and demonstrates how it is employed in big‑data platforms such as HBase and Apache Druid to achieve high‑throughput writes and scalable query processing.
This article provides a comprehensive overview of Apache Hudi 1.0, detailing its architectural redesign, five major development directions, and the most important new capabilities such as LSM‑tree timeline, function indexes, file‑group readers/writers, partial updates, and non‑blocking concurrency control, along with performance evaluations and resource links.
This article explains the Log-Structured Merge Tree (LSM-Tree) architecture, describing its immutable storage design, the roles of WAL, MemTable, ImmuTable, and SSTable, and detailing the write workflow, compaction process, and the associated read, space, and write amplification challenges.
This article explains LSM‑Tree based HBase compaction concepts, compares Minor and Major compactions, and shares practical tuning steps—including disabling automatic major compactions, controlling merge size, leveraging off‑peak windows, and improving merge efficiency—to reduce I/O, CPU usage, and latency in production environments.
CStore is a Rust‑implemented native graph storage engine designed for large‑scale graph analysis, supporting petabyte‑level data, offering array‑plus‑linked‑list storage, multi‑level indexing, efficient compaction, and providing detailed build instructions and future roadmap for open‑source development.
The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH mapping service uses these capabilities to store billions of registerId‑to‑ClientId mappings with high‑concurrency, low‑cost, fault‑tolerant performance across multiple replicated servers.
This article provides an in‑depth overview of Apache Paimon as a streaming lakehouse, explains its core features, file layout, consistency guarantees, and offers detailed guidance on integrating and tuning Paimon with Apache Flink for both write and read performance, multi‑writer concurrency, table management, and bucket rescaling.
The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH MappingTransformServer uses these capabilities with C++ code to store billions of registerId‑to‑ClientId mappings across multiple replicated servers for high‑concurrency, low‑latency, and fast service expansion.
The article investigates why pure‑read QPS drops significantly after bulk updates in OceanBase, reproduces the issue with a sysbench workload, analyses flame‑graph and SQL audit data, explains the LSM‑Tree read‑amplification mechanism, and proposes practical mitigation steps such as major freeze, plan binding, index creation, and the queuing‑table feature.
This article explains the LSM‑Tree foundation of OceanBase, details its tiered and leveled compaction strategies, and presents two experiments that observe Mini and Minor compactions under different configuration parameters, revealing how minor freeze and trigger settings affect data movement between L0 and L1 layers.
This article examines the fundamental differences between B‑Tree and LSM‑Tree storage structures in distributed and relational databases, detailing their write and read paths, performance trade‑offs, update handling, lock conflicts, and high‑availability considerations to help engineers choose the right engine for their workloads.
LSM‑Tree stores writes in an in‑memory MemTable then flushes ordered SSTables to disk, using Bloom filters and indexes to speed reads, while periodic compactions merge files; modern systems such as LevelDB, HBase, and ClickHouse adopt this design to achieve high write throughput despite slower point and range queries and occasional compaction overhead.
Flink Table Store, an Apache Flink subproject, provides a unified stream‑batch storage layer with SQL‑based table APIs, addressing real‑time and offline data needs, detailing its design goals, usage patterns, architectural layers, implementation choices, and upcoming roadmap.
This article explains the fundamentals of B+ trees, introduces log‑structured merge (LSM) trees as a modern alternative for write‑intensive workloads, and demonstrates how HBase leverages LSM trees—including MemStore, HFile, compaction, and Bloom filters—to achieve efficient storage and retrieval in NoSQL environments.
To meet Cloud Music’s massive algorithm‑feature KV storage needs, the team built RDB—a RocksDB‑based engine within Tair—adding bulk‑load, dual‑version imports, KV‑separation, in‑place sequence appends and protobuf field updates, cutting storage cost, write amplification and latency while scaling to billions of records and millions of QPS.
This article explains the fundamentals of Log‑Structured Merge (LSM) Trees, covering their write‑ahead log, memtable, SSTable architecture, compaction processes, read/write optimizations, and popular open‑source implementations such as LevelDB, RocksDB, and GoLevelDB.
This article compares MySQL and HBase across architecture, storage engine, indexing structures (B+ tree vs LSM tree), data access features, and ecosystem integration, highlighting each system's strengths, limitations, and the scenarios where HBase is a suitable complement to MySQL for large‑scale data workloads.
This article compares MySQL and HBase by examining their architectural designs, storage engines (B‑Tree vs LSM‑Tree), performance characteristics, ecosystem features such as TTL and multi‑versioning, and identifies scenarios where HBase is a suitable complement to MySQL for large‑scale data workloads.
This article compares MySQL and HBase by examining their architectural designs, storage engines, data access patterns, and ecosystem features, highlighting the strengths and trade‑offs of each system and outlining the scenarios where HBase is a suitable complement to MySQL.
This article compares MySQL and HBase across architecture, engine design, indexing structures, and feature sets such as TTL and multi‑versioning, highlighting how MySQL excels in low‑latency online transactions while HBase offers distributed scalability and write‑optimized storage for big‑data scenarios.
This article reviews the KVSSD paper presented at DATE 2018, explaining how close integration of LSM trees with the flash translation layer reduces write amplification, outlines the design optimizations such as K2P mapping, remapping compaction, hot‑cold separation, and discusses performance results and industry progress.
This article explores the evolution and current landscape of time‑series databases, detailing design principles, storage structures such as B‑Tree, B+Tree, and LSM‑Tree, and providing an in‑depth comparison of OpenTSDB, InfluxDB, and the emerging Apache IoTDB, while also discussing practical deployment considerations and industry use cases.
This article explains the fundamentals of LSM‑Tree storage engines, introduces SSTable concepts, describes how to build and maintain ordered log files, and provides a complete Java implementation of a tiny key‑value store with code examples and design analysis.
This article reviews the rapid growth of global data, examines the evolving landscape and classification of time‑series databases, analyzes storage engine designs such as B‑Tree versus LSM‑Tree, discusses query optimization and real‑time analytics, and outlines practical application scenarios in IoT and industrial settings.
TXRocks is a transactional storage engine based on RocksDB that offers significant space savings compared to InnoDB while maintaining comparable performance, with optimizations including sum operator pushdown and AEP-based secondary caching to address RocksDB's read performance limitations.
This article explains the fundamentals and drawbacks of B+ trees, introduces the Log-Structured Merge Tree (LSM Tree) used in HBase and other NoSQL databases, and discusses how LSM architecture, Bloom filters, and MemStore improve write performance while affecting read efficiency.
Since 2019 NetEase Media has migrated several recommendation and account services from RDS to MyRocks, cutting disk usage by up to 68 % and halving response times while handling 40‑50 k QPS write‑heavy workloads, though the engine lacks partitioning, online DDL, and certain index types, requiring careful workload assessment.
LevelDB stores data using an in‑memory Memtable that flushes to immutable tables and disk‑based SSTables, writes are logged then batched and applied through a writer queue, reads check Memtable, immutable Memtable, then SSTables, and background compactions merge tables to improve read performance and reclaim space.
This article explains the principles, evolution, modern structures, compaction strategies, optimization techniques such as Bloom filters and partitioning, and concurrency and recovery mechanisms of Log-Structured Merge (LSM) trees, which are widely used in contemporary NoSQL storage systems.
This article explains OceanBase's public‑cloud deployment, its unique architecture without a central controller, horizontal scaling via partition groups, LSM‑Tree storage design, advanced SQL engine features, ACID‑plus‑Availability guarantees, and real‑world performance records, illustrating why it suits high‑availability financial workloads.
This article explains how OceanBase’s distributed, shared‑nothing architecture, with dual data replicas, Paxos‑based consistency, online compression, and resource‑isolated compaction, enables it to achieve top TPC‑C performance while addressing storage costs and CPU usage.
This article introduces CB‑SQL, a MySQL‑compatible elastic database built on CockroachDB and RocksDB, explains RocksDB’s LSM‑tree design, details key configuration parameters, and describes the read/write/clear/file‑operation optimizations employed by CB‑SQL to achieve high scalability and stability.
The article explains why HBase read operations are complex due to its LSM‑Tree storage and multi‑version design, details the step‑by‑step Scan workflow, discusses the reasons for its multi‑request architecture, compares scan performance with Parquet and Kudu, and offers recommendations for large‑scale data scanning.
This article presents the design, implementation, and evaluation of X‑Engine, a next‑generation LSM‑Tree based storage engine that offloads compaction to FPGA, achieving up to 50% KV‑interface and 40% SQL‑interface performance gains for write‑intensive OLTP workloads.
This article introduces Alibaba’s X‑Engine storage engine, the foundation of the next‑generation distributed database X‑DB, and explains how FPGA‑accelerated compaction and asynchronous scheduling dramatically improve write‑intensive OLTP performance, reduce CPU contention, and achieve up to 50 % throughput gains while maintaining fault tolerance.
This article surveys current storage research and industry developments, covering Open‑Channel SSD architecture, non‑volatile memory programming models, machine‑learning‑driven learned indexes, LSM‑Tree performance improvements, crash‑consistency verification, and recent virtualization and container‑storage advances.
LevelDB, Google's high-performance key‑value store built on LSM trees, uses an in‑memory skip‑list, immutable memtables, and sstable files organized in multi‑level compaction, offering interfaces for creation, reads, writes, snapshots, and new features like fuzzy search and JSON storage, all explained with diagrams.
This article explains the fundamentals, use cases, challenges, and storage architectures of time‑series databases, illustrating how they handle massive IoT telemetry through log‑structured storage, sharding, and distributed designs while supporting high‑throughput writes and fast analytical queries.
