Tagged articles
LSM-tree
57 articles · Page 1 of 1
This article dissects VictoriaLogs' performance engineering across four layers—data architecture (LSM-tree, time partitioning, log-stream locality), query optimization (sparse indexes, Bloom filters), columnar storage with type-specific encoding and compression, and Go runtime tuning—showing that reducing unnecessary work via smart data organization outweighs micro-optimizations.
Bloom FilterGo performanceLSM-tree
0 likes · 26 min read

This article analyzes two financial risk control lakehouse architectures: DiDi uses StarRocks to unify real-time and batch feature serving with primary-key tables and bucket pruning, while Ant Group replaces HBase and Elasticsearch with Apache Paimon's first-row, partial-update, and sort-compaction mechanisms for long-window deduplication, fast reconciliation, and historical lookback.
Apache PaimonLSM-treeOLAP
0 likes · 23 min read

The article analyzes two online travel case studies: Tongcheng Travel migrating from Hudi to Paimon to solve slow large-table real-time writes using LSM-tree storage, and Trip.com adopting StarRocks with materialized views to unify lake queries and eliminate data warehouse copies, detailing migration methodologies, architectural reasoning, and production troubleshooting.
Apache HudiApache PaimonLSM-tree
0 likes · 22 min read

This article deep-dives into Apache Paimon's kernel source code, explaining the LSM write path from memory buffer to Sorted Runs, compaction scheduling trade-offs, four changelog production modes, and two-phase commit with Flink checkpoints, providing troubleshooting guidance for compaction backlog, changelog latency, and snapshot bloat.
Apache PaimonFlinkLSM-tree
0 likes · 33 min read

This article traces the origin of the Log-Structured Merge-Tree (LSM Tree) from a 1996 paper by Patrick O'Neil et al., explaining how its sequential-write, background-compaction design solved the random I/O bottleneck of B-Trees, enabling modern write-heavy systems like Cassandra, RocksDB, and Kafka.
B+TreeBigtableDatabase Storage
0 likes · 16 min read

360 Zhihui Cloud's ZestKV is a Redis-compatible, cloud-native serverless KV store that separates compute from storage using shared object storage and a log service, enabling elastic scaling via region sharding, background task decoupling, and a dual-async write path achieving 700K SET QPS in benchmarks.
Cloud-Native DatabaseElastic ScalingLSM-tree
0 likes · 24 min read

Apache Paimon combines a lakehouse table format with an LSM-tree update engine to handle continuous updates, historical snapshots, and multi-engine access in real-time data warehouses, reducing the need for separate Kafka, Hive, and OLAP systems.
Apache PaimonCDCFlink
0 likes · 23 min read

This comprehensive guide covers Bitmap index internals, cardinality thresholds, index replacement rules, and layered optimization combining partition pruning, BloomFilter, and Range-Bitmap for StarRocks querying Paimon tables, with official binary formats, production thresholds, and real-world case studies.
Apache PaimonBitmap IndexBloomFilter
0 likes · 26 min read

The article details JD’s data lake architecture, its 500 PB scale, self‑developed Hudi extensions—including LSM‑Tree‑based MoR tables, custom indexing, IO optimizations, Flink stream scheduling, and NativeIO SDK—along with benchmarks, community contributions, and future roadmap for real‑time big‑data processing.
HudiLSM-treebig data
0 likes · 19 min read

This article explains how to improve Paimon's query performance by optimizing table schemas, storage settings, query parameters, and index designs, covering table mode choices, partitioning, file formats, parallelism, batch reads, and various index types such as Bloom filters and clustering indexes.
LSM-treePaimonSQL
0 likes · 8 min read


21CTOApr 20, 2025 · Databases
This article walks you through the inner workings of modern databases—explaining storage engines, query parsing, execution, and transaction models—while comparing classic B‑Tree structures with newer LSM‑Tree designs to help you decide whether SQL or NoSQL best fits your performance and consistency needs.
B+TreeLSM-treeStorage Engines
0 likes · 8 min read

The article walks through a complete Zig implementation of an LSM‑Tree, detailing its in‑memory skip‑list MemTable, immutable SSTable blocks with compression and Bloom filters, write‑ahead logging, iterator hierarchy for reads, and multi‑level compaction logic that merges and rewrites SSTables.
LSM-treeZigcompaction
0 likes · 42 min read

This article provides a comprehensive overview of Apache Paimon, covering its real‑time lake ingestion, unified stream‑batch processing, table types (primary‑key and append‑only), LSM‑tree storage, bucket mechanisms, merge‑engine options, compaction strategies, concurrency control, consumption methods, tag management, data cleanup, and system tables for big‑data workloads.
Apache PaimonFlinkLSM-tree
0 likes · 25 min read

This article walks through building a database storage engine from a simple shell script to a full distributed key‑value system, covering in‑memory indexing, SSTable creation, LSM‑Tree architecture with compaction, replication strategies, and sharding techniques for scaling across multiple machines.
B+TreeData ShardingDistributed Database
0 likes · 38 min read

RocksDB, an open‑source key‑value storage engine created by Facebook in 2012, delivers high performance, durability, and tunable configurations for flash and SSDs; the article details its core features, internal components such as MemTable and Bloom filters, and real‑world uses in TiDB and Apache Flink.
Database EngineFlinkLSM-tree
0 likes · 10 min read

The article explains Apache Hudi’s three‑layer architecture and details four major 1.0 enhancements—LSM‑tree timeline, non‑blocking concurrency control, file‑group reader/writer APIs, and function indexes—while providing a brief review and links to the Hudi 1.x RFC.
Apache HudiFunction IndexLSM-tree
0 likes · 9 min read

Modern databases have largely shifted from B‑tree based storage to LSM‑tree engines due to SSD hardware characteristics, high‑write workloads, concurrency advantages, simpler implementation, and evolving application demands, with additional insights into Paxos/Raft consensus, common database jargon, and performance optimizations.
ConcurrencyDatabase JargonDatabase Storage
0 likes · 11 min read

This article explains Paimon's core concepts—including snapshots, partitions, buckets, consistency guarantees, file layout, LSM‑tree organization, and compaction strategies—while also covering table management tasks such as snapshot expiration, rollback, partition expiration, and small‑file mitigation techniques.
BucketsLSM-treePaimon
0 likes · 12 min read

The article explains the Log-Structured Merge-Tree (LSM) architecture, its core components, advantages and disadvantages, and demonstrates how it is employed in big‑data platforms such as HBase and Apache Druid to achieve high‑throughput writes and scalable query processing.
LSM-treeStoragecompaction
0 likes · 7 min read

This article provides a comprehensive overview of Apache Hudi 1.0, detailing its architectural redesign, five major development directions, and the most important new capabilities such as LSM‑tree timeline, function indexes, file‑group readers/writers, partial updates, and non‑blocking concurrency control, along with performance evaluations and resource links.
Apache HudiFunction IndexLSM-tree
0 likes · 14 min read

This article explains the Log-Structured Merge Tree (LSM-Tree) architecture, describing its immutable storage design, the roles of WAL, MemTable, ImmuTable, and SSTable, and detailing the write workflow, compaction process, and the associated read, space, and write amplification challenges.
LSM-treeLog-Structured Merge TreeMemtable
0 likes · 7 min read

This article explains LSM‑Tree based HBase compaction concepts, compares Minor and Major compactions, and shares practical tuning steps—including disabling automatic major compactions, controlling merge size, leveraging off‑peak windows, and improving merge efficiency—to reduce I/O, CPU usage, and latency in production environments.
HBaseLSM-treePerformance Tuning
0 likes · 11 min read

CStore is a Rust‑implemented native graph storage engine designed for large‑scale graph analysis, supporting petabyte‑level data, offering array‑plus‑linked‑list storage, multi‑level indexing, efficient compaction, and providing detailed build instructions and future roadmap for open‑source development.
CStoreIndexingLSM-tree
0 likes · 12 min read

The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH mapping service uses these capabilities to store billions of registerId‑to‑ClientId mappings with high‑concurrency, low‑cost, fault‑tolerant performance across multiple replicated servers.
Column FamilyLSM-treeMessage Push
0 likes · 24 min read

This article walks through building a simple yet enterprise‑grade key‑value storage engine in Go, covering WAL implementation, LSM‑Tree structure, SkipList‑based in‑memory storage, transaction handling for Put/Get/Delete, and the on‑disk log format, with a complete 1,000‑line codebase hosted on GitHub.
GoKV storeLSM-tree
0 likes · 6 min read

This article provides an in‑depth overview of Apache Paimon as a streaming lakehouse, explains its core features, file layout, consistency guarantees, and offers detailed guidance on integrating and tuning Paimon with Apache Flink for both write and read performance, multi‑writer concurrency, table management, and bucket rescaling.
Apache PaimonFlinkLSM-tree
0 likes · 23 min read

The article explains RocksDB’s LSM‑based architecture, column‑family isolation, and snapshot features, and shows how Vivo’s VPUSH MappingTransformServer uses these capabilities with C++ code to store billions of registerId‑to‑ClientId mappings across multiple replicated servers for high‑concurrency, low‑latency, and fast service expansion.
Column FamilyLSM-treeMessage Push
0 likes · 25 min read

The article investigates why pure‑read QPS drops significantly after bulk updates in OceanBase, reproduces the issue with a sysbench workload, analyses flame‑graph and SQL audit data, explains the LSM‑Tree read‑amplification mechanism, and proposes practical mitigation steps such as major freeze, plan binding, index creation, and the queuing‑table feature.
Flame GraphLSM-treeMajor Freeze
0 likes · 16 min read

This article explains the LSM‑Tree foundation of OceanBase, details its tiered and leveled compaction strategies, and presents two experiments that observe Mini and Minor compactions under different configuration parameters, revealing how minor freeze and trigger settings affect data movement between L0 and L1 layers.
Database StorageLSM-treeMini Compaction
0 likes · 13 min read

This article examines the fundamental differences between B‑Tree and LSM‑Tree storage structures in distributed and relational databases, detailing their write and read paths, performance trade‑offs, update handling, lock conflicts, and high‑availability considerations to help engineers choose the right engine for their workloads.
B-TreeDatabase StorageLSM-tree
0 likes · 25 min read

LSM‑Tree stores writes in an in‑memory MemTable then flushes ordered SSTables to disk, using Bloom filters and indexes to speed reads, while periodic compactions merge files; modern systems such as LevelDB, HBase, and ClickHouse adopt this design to achieve high write throughput despite slower point and range queries and occasional compaction overhead.
Bloom FilterClickHouseHBase
0 likes · 11 min read

Flink Table Store, an Apache Flink subproject, provides a unified stream‑batch storage layer with SQL‑based table APIs, addressing real‑time and offline data needs, detailing its design goals, usage patterns, architectural layers, implementation choices, and upcoming roadmap.
FlinkLSM-treeStorage
0 likes · 14 min read

This article explains the fundamentals of B+ trees, introduces log‑structured merge (LSM) trees as a modern alternative for write‑intensive workloads, and demonstrates how HBase leverages LSM trees—including MemStore, HFile, compaction, and Bloom filters—to achieve efficient storage and retrieval in NoSQL environments.
B+TreeHBaseLSM-tree
0 likes · 7 min read

To meet Cloud Music’s massive algorithm‑feature KV storage needs, the team built RDB—a RocksDB‑based engine within Tair—adding bulk‑load, dual‑version imports, KV‑separation, in‑place sequence appends and protobuf field updates, cutting storage cost, write amplification and latency while scaling to billions of records and millions of QPS.
Algorithm FeaturesKV SeparationKV storage
0 likes · 16 min read

This article explains the fundamentals of Log‑Structured Merge (LSM) Trees, covering their write‑ahead log, memtable, SSTable architecture, compaction processes, read/write optimizations, and popular open‑source implementations such as LevelDB, RocksDB, and GoLevelDB.
LSM-treeLevelDBRocksDB
0 likes · 15 min read

This article compares MySQL and HBase across architecture, storage engine, indexing structures (B+ tree vs LSM tree), data access features, and ecosystem integration, highlighting each system's strengths, limitations, and the scenarios where HBase is a suitable complement to MySQL for large‑scale data workloads.
B+TreeHBaseLSM-tree
0 likes · 9 min read

This article compares MySQL and HBase by examining their architectural designs, storage engines (B‑Tree vs LSM‑Tree), performance characteristics, ecosystem features such as TTL and multi‑versioning, and identifies scenarios where HBase is a suitable complement to MySQL for large‑scale data workloads.
B+TreeDatabase ArchitectureHBase
0 likes · 8 min read

This article compares MySQL and HBase by examining their architectural designs, storage engines, data access patterns, and ecosystem features, highlighting the strengths and trade‑offs of each system and outlining the scenarios where HBase is a suitable complement to MySQL.
B+TreeHBaseLSM-tree
0 likes · 5 min read

This article compares MySQL and HBase across architecture, engine design, indexing structures, and feature sets such as TTL and multi‑versioning, highlighting how MySQL excels in low‑latency online transactions while HBase offers distributed scalability and write‑optimized storage for big‑data scenarios.
B+TreeDatabase ArchitectureHBase
0 likes · 9 min read

This article reviews the KVSSD paper presented at DATE 2018, explaining how close integration of LSM trees with the flash translation layer reduces write amplification, outlines the design optimizations such as K2P mapping, remapping compaction, hot‑cold separation, and discusses performance results and industry progress.
KVSSDLSM-treeNVMe
0 likes · 15 min read

This article explores the evolution and current landscape of time‑series databases, detailing design principles, storage structures such as B‑Tree, B+Tree, and LSM‑Tree, and providing an in‑depth comparison of OpenTSDB, InfluxDB, and the emerging Apache IoTDB, while also discussing practical deployment considerations and industry use cases.
Apache IoTDBB+TreeInfluxDB
0 likes · 38 min read

This article explains the fundamentals of LSM‑Tree storage engines, introduces SSTable concepts, describes how to build and maintain ordered log files, and provides a complete Java implementation of a tiny key‑value store with code examples and design analysis.
Database InternalsJavaLSM-tree
0 likes · 17 min read

This article reviews the rapid growth of global data, examines the evolving landscape and classification of time‑series databases, analyzes storage engine designs such as B‑Tree versus LSM‑Tree, discusses query optimization and real‑time analytics, and outlines practical application scenarios in IoT and industrial settings.
IoTLSM-treeTime Series Database
0 likes · 19 min read

TXRocks is a transactional storage engine based on RocksDB that offers significant space savings compared to InnoDB while maintaining comparable performance, with optimizations including sum operator pushdown and AEP-based secondary caching to address RocksDB's read performance limitations.
AEP cachingInnoDB comparisonLSM-tree
0 likes · 15 min read

This article explains the fundamentals and drawbacks of B+ trees, introduces the Log-Structured Merge Tree (LSM Tree) used in HBase and other NoSQL databases, and discusses how LSM architecture, Bloom filters, and MemStore improve write performance while affecting read efficiency.
B+TreeBloom FilterHBase
0 likes · 6 min read

Since 2019 NetEase Media has migrated several recommendation and account services from RDS to MyRocks, cutting disk usage by up to 68 % and halving response times while handling 40‑50 k QPS write‑heavy workloads, though the engine lacks partitioning, online DDL, and certain index types, requiring careful workload assessment.
LSM-treeMyRocksRocksDB
0 likes · 12 min read

LevelDB stores data using an in‑memory Memtable that flushes to immutable tables and disk‑based SSTables, writes are logged then batched and applied through a writer queue, reads check Memtable, immutable Memtable, then SSTables, and background compactions merge tables to improve read performance and reclaim space.
Database InternalsLSM-treeLevelDB
0 likes · 16 min read

This article explains the principles, evolution, modern structures, compaction strategies, optimization techniques such as Bloom filters and partitioning, and concurrency and recovery mechanisms of Log-Structured Merge (LSM) trees, which are widely used in contemporary NoSQL storage systems.
Bloom FilterLSM-treeNoSQL
0 likes · 12 min read

This article explains OceanBase's public‑cloud deployment, its unique architecture without a central controller, horizontal scaling via partition groups, LSM‑Tree storage design, advanced SQL engine features, ACID‑plus‑Availability guarantees, and real‑world performance records, illustrating why it suits high‑availability financial workloads.
High AvailabilityLSM-treeOceanBase
0 likes · 12 min read

This article explains how OceanBase’s distributed, shared‑nothing architecture, with dual data replicas, Paxos‑based consistency, online compression, and resource‑isolated compaction, enables it to achieve top TPC‑C performance while addressing storage costs and CPU usage.
Distributed DatabaseLSM-treeOceanBase
0 likes · 10 min read

This article introduces CB‑SQL, a MySQL‑compatible elastic database built on CockroachDB and RocksDB, explains RocksDB’s LSM‑tree design, details key configuration parameters, and describes the read/write/clear/file‑operation optimizations employed by CB‑SQL to achieve high scalability and stability.
CB-SQLDatabase ArchitectureLSM-tree
0 likes · 13 min read

The article explains why HBase read operations are complex due to its LSM‑Tree storage and multi‑version design, details the step‑by‑step Scan workflow, discusses the reasons for its multi‑request architecture, compares scan performance with Parquet and Kudu, and offers recommendations for large‑scale data scanning.
HBaseLSM-treeSCAN
0 likes · 7 min read

This article presents the design, implementation, and evaluation of X‑Engine, a next‑generation LSM‑Tree based storage engine that offloads compaction to FPGA, achieving up to 50% KV‑interface and 40% SQL‑interface performance gains for write‑intensive OLTP workloads.
FPGALSM-treeX-Engine
0 likes · 19 min read

This article introduces Alibaba’s X‑Engine storage engine, the foundation of the next‑generation distributed database X‑DB, and explains how FPGA‑accelerated compaction and asynchronous scheduling dramatically improve write‑intensive OLTP performance, reduce CPU contention, and achieve up to 50 % throughput gains while maintaining fault tolerance.
FPGALSM-treeX-Engine
0 likes · 21 min read

This article surveys current storage research and industry developments, covering Open‑Channel SSD architecture, non‑volatile memory programming models, machine‑learning‑driven learned indexes, LSM‑Tree performance improvements, crash‑consistency verification, and recent virtualization and container‑storage advances.
LSM-treeLearned IndexNVM
0 likes · 18 min read

LevelDB, Google's high-performance key‑value store built on LSM trees, uses an in‑memory skip‑list, immutable memtables, and sstable files organized in multi‑level compaction, offering interfaces for creation, reads, writes, snapshots, and new features like fuzzy search and JSON storage, all explained with diagrams.
Bloom FilterDatabase ArchitectureLSM-tree
0 likes · 11 min read

This article explains the fundamentals, use cases, challenges, and storage architectures of time‑series databases, illustrating how they handle massive IoT telemetry through log‑structured storage, sharding, and distributed designs while supporting high‑throughput writes and fast analytical queries.
IoTLSM-treeSharding
0 likes · 14 min read
