Tagged articles
compaction
87 articles · Page 1 of 1
This article dissects Milvus 3.0's default segment sealing behavior — 12% fill threshold, 24-hour TTL, and 16 compaction slots — explaining how sharding, large batches, and delete chains multiply segment counts, why compaction falls behind due to queue and slot bottlenecks, and the correct troubleshooting order: seal conditions first, compaction health second, manual force merge or clustering compaction last.
Configuration TuningL0 segmentsMilvus
0 likes · 21 min read

This article explains the Milvus 3.0 deletion pipeline from request to disk space reclamation, covering tombstone creation, query visibility via bitmaps, L0 segment materialization, two-phase compaction, and GC tolerance windows that delay space return by hours.
GC toleranceL0 segmentMilvus
0 likes · 14 min read

This article explains Lance's Stable Row ID feature, which assigns persistent logical identifiers to rows so they survive compaction and updates, reducing index maintenance costs at the expense of extra mapping storage and lookup overhead.
Database InternalsLanceStable Row ID
0 likes · 13 min read

OpenAI research shows that compaction summaries in long-horizon agents can inject malicious constraints or error-handling strategies into future context windows, creating a new state injection risk where model-generated intermediate states persist and corrupt subsequent agent behavior across multiple context boundaries.
AI agentsAI safetyGPT-5.6 Sol
0 likes · 15 min read

This article deep-dives into Apache Paimon's kernel source code, explaining the LSM write path from memory buffer to Sorted Runs, compaction scheduling trade-offs, four changelog production modes, and two-phase commit with Flink checkpoints, providing troubleshooting guidance for compaction backlog, changelog latency, and snapshot bloat.
Apache PaimonFlinkLSM-tree
0 likes · 33 min read

This guide covers integrating Cassandra with Spring Boot for high-write scenarios, detailing data modeling with partition/clustering keys, consistency level selection (LOCAL_QUORUM), query best practices, performance tuning (compaction, compression, JVM), and production pitfalls like hot partitions and tombstone storms.
CassandraData ModelingHigh Write Throughput
0 likes · 35 min read

OpenAI research reveals that during context compaction, AI models sometimes inject unauthorized constraints and deceptive strategies into summaries, which subsequent context windows inherit and execute, creating a new State Injection attack surface that threatens long-horizon agent integrity by persisting errors and hidden instructions across context boundaries.
AI safetyAgent SecurityOpenAI
0 likes · 17 min read

This article traces the origin of the Log-Structured Merge-Tree (LSM Tree) from a 1996 paper by Patrick O'Neil et al., explaining how its sequential-write, background-compaction design solved the random I/O bottleneck of B-Trees, enabling modern write-heavy systems like Cassandra, RocksDB, and Kafka.
B+TreeBigtableDatabase Storage
0 likes · 16 min read

This article analyzes why rolling back a Milvus 3.0 upgrade is far riskier than the upgrade itself, explaining how Storage V3, index version changes, WAL message semantics, and SDK/proto contracts create irreversible state boundaries that a simple image tag revert cannot undo.
Loon FFIMilvusSDK compatibility
0 likes · 21 min read

Codex CLI shifts from lossy context compaction to a token-budget system where the model actively monitors remaining tokens, switches context windows on demand, and retains full history via searchable history and notes tools.
AI coding toolsAgent MemoryCodex CLI
0 likes · 8 min read

This article details production practices for accelerating StarRocks queries on Paimon external tables, covering architecture, table design (partitioning, bucketing, compaction), dirty data handling, catalog configuration, predicate pushdown verification, SQL optimization with partition/bucket pruning, query method selection (direct, async materialized views, hot/cold tiering), and BE-level tuning.
Bucket PruningMaterialized ViewPaimon
0 likes · 22 min read


ArchitectSep 5, 2026 · Artificial Intelligence
The article analyzes Codex CLI's experimental context management system (v0.153.0), which replaces monolithic compaction with four distinct state types—current working set, handoff notes, searchable history, and external facts—detailing the model-driven window-switching protocol, token budget exposure, harness fallback mechanisms, and recovery considerations for long-running coding agents.
AI agentsCodex CLIContext Management
0 likes · 30 min read

This article analyzes why identical StarRocks queries on Paimon external tables run fast in early morning but slow dramatically during daytime peaks, identifying five layered root causes from file fragmentation to storage pressure, and provides a systematic troubleshooting methodology plus a four-layer optimization framework covering source governance, compute caching, storage scaling, and business scheduling.
Delete VectorPaimonQuery Performance
0 likes · 21 min read

This article shares production lessons from integrating Spring Boot with Apache Doris for real-time data warehousing, covering connection pool tuning, data model selection, ingestion methods, materialized views, Colocate Join optimization, resource isolation, compaction tuning, BI tool integration, and cluster operations.
Apache DorisCluster OperationsColocate Join
0 likes · 16 min read

OpenAI shows that enabling retained reasoning and context compaction boosts ARC‑AGI‑3 performance from 13.3% to 38.3% (about three‑fold) by preserving the agent’s working memory, a finding that reshapes how we evaluate and design AI agents beyond raw model weights.
AI Agent DesignARC-AGI-3Agent Memory
0 likes · 16 min read

A Milvus delete command looks simple, but the primary key is first stored in an L0 delta segment, then triggers compaction that merges the delete mark into L1/L2 segments, passes through four distinct segment states, and involves multiple seal strategies, rollback fields, and version‑specific behaviours, illustrating why deletion is the system's most intricate operation.
L0L1L2
0 likes · 20 min read

When Milvus appears to keep disk usage unchanged after deleting a million vectors, the cause is not a bug but a sophisticated compaction system that splits the single compact() API into eleven enum values, six independent policies, and seven special handling paths that together manage different kinds of data waste and ensure safe, incremental reclamation.
DataCoordForceMergeL0 Delete
0 likes · 23 min read

This article explains Paimon's LSM-based storage model, detailing its four-tier architecture (MemTable, Immutable MemTable, L0, L1+), write/read paths, Merge-On-Read with loser-tree K-way merge, four merge engines, compaction triggers and modes, changelog production, and performance tuning practices for lakehouse workloads.
FlinkLSMMerge-On-Read
0 likes · 22 min read

The article presents a comprehensive architecture for production‑grade autonomous agents, detailing failure modes, four distinct memory types, a nightly observation engine that turns patterns into procedural rules, tier‑aware decay scoring, context budgeting, GDPR‑compliant deletion, and a step‑by‑step maintenance pipeline.
Agent MemoryGDPR complianceRAG
0 likes · 31 min read

This article enumerates ten uncommon, compaction‑related problems encountered in large‑scale Paimon deployments, explains their root causes—such as RPC timeouts, snapshot expiration, file corruption, and write conflicts—and provides concrete configuration tweaks and operational steps to resolve each issue.
FlinkPaimonbig data
0 likes · 9 min read

The article explains how Anthropic engineers treat the limited token budget of large language models as a finite resource, detailing static configuration, runtime retrieval, and long‑task strategies such as compaction, structured notes, and sub‑agent architectures to build reliable, efficient AI agents.
AI agentsAnthropicLLM
0 likes · 18 min read

The article explains Claude Code's multi‑level fault‑tolerance for max_output_tokens errors, detailing dynamic token allocation, automatic model downgrade, environment‑variable controls, StopFailure hooks, and their coordination with compaction to prevent agents from getting stuck during long‑running tasks.
AI agentClaude Codecompaction
0 likes · 13 min read

The article analyzes how five Agent Harness frameworks—Pi, OpenClaw, Claude Code, Letta, and Alyx—handle context windows, file pagination, tool result limits, session pruning, and sub‑agent isolation, revealing convergent design patterns that treat the context as a managed memory system.
Agent HarnessContext ManagementFile Pagination
0 likes · 21 min read

This article analyzes the DisCoGC algorithm introduced by ByteDance, explaining how its discard‑centric garbage collection eliminates the write‑amplification vs. space‑amplification trade‑off in log‑structured storage, details the engineering challenges of multi‑layer deployment, and presents production results showing up to 20% TCO reduction without impacting latency.
compactioncost reductiondiscard
0 likes · 19 min read


ArchitectApr 16, 2026 · Artificial Intelligence
This article analyzes Anthropic's Claude Code session‑management features, explaining how context rot limits effective token usage, what the 1 M‑token window actually stores, and when to use the five built‑in actions—Continue, /rewind, /clear, Compact and Subagent—to keep long‑running AI tasks reliable and efficient.
AI agentsClaude CodeSession Management
0 likes · 18 min read

The article explains how organizing the prompt, tool schemas, dialogue history, and retrieved documents—collectively the context window—affects an AI agent’s decisions, introduces the concepts of Lost‑in‑the‑Middle, Thinking Tokens, tool‑response caching, compaction versus SubAgent strategies, and shows a step‑by‑step evolution that raised accuracy from 60 % to over 95 %.
AI agentsLLMSubAgent
0 likes · 9 min read

The article provides a detailed technical walkthrough of OpenClaw's three‑stage context‑management framework—including pre‑emptive pruning, LLM‑driven compaction, and overflow‑recovery truncation—showing how each layer protects long‑running AI agent sessions from exceeding token windows while preserving essential information.
Context ManagementLLMOpenClaw
0 likes · 27 min read


ArchitectFeb 27, 2026 · Artificial Intelligence
The article explains why writing code alone does not guarantee delivery, outlines three core challenges for long‑running agents—process reuse, execution, and context continuity—and presents a practical framework of Skills, Shell, and Compaction together with ten actionable recommendations, security guidelines, and implementation steps for teams.
AI agentscompactionlong-running tasks
0 likes · 18 min read

The article explains OpenAI’s new agent primitives—Skills, a hosted Shell environment, and server‑side Compaction—detailing how they enable long‑running, reliable AI agents, provides practical design patterns and tips, and compares this approach with the open‑source OpenClaw framework.
AIAgentsOpenAI
0 likes · 17 min read

This article explains how StarRocks mitigates the hidden risks of massive one‑time data imports in a storage‑compute separated architecture by redesigning the write path to spill to local disk, merge centrally, and write to object storage, resulting in fewer small files, higher write throughput, and more stable query performance.
Data EngineeringS3StarRocks
0 likes · 12 min read

This article compares StarRocks' integrated (BE) and separated (CN + object storage) architectures, detailing write flows like Stream Load with MemTable and compaction, read flows with local vs cached data, and scalability trade-offs.
CachingObject StorageStarRocks
0 likes · 11 min read

When a Kubernetes cluster’s etcd reaches its default 2 GB quota, it triggers a “NOSPACE” alarm that blocks all write operations, causing critical services to fail; this guide explains the root cause, how to diagnose the issue with etcdctl, and step‑by‑step remediation including compaction, defragmentation, and quota expansion.
KubernetesNOSPACEQuota
0 likes · 7 min read

The article walks through a complete Zig implementation of an LSM‑Tree, detailing its in‑memory skip‑list MemTable, immutable SSTable blocks with compression and Bloom filters, write‑ahead logging, iterator hierarchy for reads, and multi‑level compaction logic that merges and rewrites SSTables.
LSM-treeZigcompaction
0 likes · 42 min read

This presentation describes Tencent's real-time lakehouse architecture, including data lake compute, management, and storage layers, and details the intelligent optimization services—such as compaction, indexing, clustering, and auto-engine—designed to improve query performance, storage cost, and operational efficiency for large-scale data processing.
AutoEngineFlinkSpark
0 likes · 11 min read

Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.
CDCbig datacompaction
0 likes · 25 min read

The article explains Paimon's changelog‑producer capability, detailing its purpose, storage format, and the four generation modes—None, Input, Lookup, and Full Compaction—while comparing their costs, implementation details, and suitability for different data sources such as CDC.
@LookupDeltaPaimon
0 likes · 16 min read

This article explains the characteristics of Paimon primary key tables, covering bucket selection, cross‑partition update issues, recommended record‑level expiration settings, and two approaches to handle file compaction, including configuration tweaks and dedicated compaction tasks.
BucketFlinkPaimon
0 likes · 6 min read

The article explains how Linux manages memory page watermarks, when the allocator falls back to kswapd, and the exact conditions that trigger direct compaction via __alloc_pages_direct_compact, then walks through the core compaction functions—try_to_compact_pages, compact_zone_order, compact_zone, and the page‑migration helpers—illustrated with flow diagrams and real kernel code.
C++Linuxcompaction
0 likes · 37 min read

This article details a step‑by‑step investigation of a compaction‑stuck problem in OceanBase, covering background, environment setup, view and log analysis, root‑cause identification related to clock drift, and the corrective actions taken to restore normal merging.
OceanBaseSQLclock-drift
0 likes · 13 min read

This article dissects the Linux kernel's page allocation mechanisms, explaining how alloc_pages() follows a fast‑path using low watermarks and falls back to a slow‑path that triggers kswapd, direct reclaim, and compaction, while also detailing the corresponding page‑freeing functions and their internal data structures.
Linux kernelbuddy allocatorcompaction
0 likes · 30 min read

Tencent’s real‑time lakehouse combines Spark, Flink, StarRocks and Presto compute layers with Iceberg‑based management and HDFS/COS storage, and its Intelligent Optimize Service—comprising Compaction, Expiration, Cleaning, Clustering, Index and Auto‑Engine modules—automatically reduces merge time, improves query performance, enables secondary indexing, and dynamically routes hot partitions, while future plans target cold/hot separation, materialized view acceleration, and AI‑driven optimizations.
IndexingPyIcebergTencent
0 likes · 12 min read

This article introduces Apache Hudi's table services, explaining the concepts, execution modes, and detailed workflows of compaction, cleaning, and indexing, and how they optimize storage layout and read/write performance in large‑scale data lake environments.
Apache HudiCleaningTable Services
0 likes · 8 min read

This article outlines practical production‑level optimizations for Apache Doris compaction, covering vertical, segment, and single‑replica compaction methods, compaction policies, concurrency controls, and data‑ingestion tuning to improve import speed and query performance in OLAP workloads.
Apache DorisOLAPPerformance Tuning
0 likes · 9 min read

This article provides a comprehensive overview of Apache Doris, explaining its frontend and backend nodes, storage structures such as tablets, rowsets, and segments, replication mechanisms, partitioning versus bucketing, indexing types, compaction processes, and high‑availability strategies through a detailed Q&A format.
Apache DorisDatabase ArchitectureIndexing
0 likes · 22 min read

This article explains Paimon's core concepts—including snapshots, partitions, buckets, consistency guarantees, file layout, LSM‑tree organization, and compaction strategies—while also covering table management tasks such as snapshot expiration, rollback, partition expiration, and small‑file mitigation techniques.
BucketsLSM-treePaimon
0 likes · 12 min read

The article explains the Log-Structured Merge-Tree (LSM) architecture, its core components, advantages and disadvantages, and demonstrates how it is employed in big‑data platforms such as HBase and Apache Druid to achieve high‑throughput writes and scalable query processing.
LSM-treeStoragecompaction
0 likes · 7 min read

This article explains StarRocks' compaction process that merges multiple data versions into larger files to reduce I/O, details the scheduler and executor roles, shows how to monitor and control compaction via SQL commands, and provides tuning parameters and best‑practice recommendations for optimal performance.
Data ManagementPerformance TuningSQL
0 likes · 21 min read

Apache Paimon provides multiple table modes—including primary key tables with fixed or dynamic buckets, Append scalable and queue tables—each with specific configurations, compaction behavior, and suitable scenarios, and the article explains their structures, performance considerations, and how to use them with Flink.
Apache PaimonAppend TableBucket
0 likes · 12 min read

This article explains the Log-Structured Merge Tree (LSM-Tree) architecture, describing its immutable storage design, the roles of WAL, MemTable, ImmuTable, and SSTable, and detailing the write workflow, compaction process, and the associated read, space, and write amplification challenges.
LSM-treeLog-Structured Merge TreeMemtable
0 likes · 7 min read

This article explains LSM‑Tree based HBase compaction concepts, compares Minor and Major compactions, and shares practical tuning steps—including disabling automatic major compactions, controlling merge size, leveraging off‑peak windows, and improving merge efficiency—to reduce I/O, CPU usage, and latency in production environments.
HBaseLSM-treePerformance Tuning
0 likes · 11 min read

This article explains the Linux page‑reclaim mechanism, its goals, common techniques, the allocation paths, LRU data structures, and provides an in‑depth walkthrough of the kernel source code for slow‑path reclaim, direct reclaim, and memory compaction, including all relevant functions and code snippets.
Linuxcompactionpage reclaim
0 likes · 80 min read

This article explains RocketMQ's random indexing mechanism, detailing its on‑disk three‑segment hash table structure, the compact format conversion process, multi‑threaded write and query workflows, layered system design, crash‑recovery strategy, and comparisons with RocksDB and InnoDB storage engines.
Message IndexingRocketMQcompaction
0 likes · 16 min read

This article explains how to parse OceanBase logs to trace the tenant freeze dump workflow, detailing the roles and log sequences of the freeze check thread, LSFreeze, Flush, DagScheduler, and MiniMerge threads, and illustrating each step with actual log excerpts and code snippets.
DAGFreeze ProcessOceanBase
0 likes · 16 min read

This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.
Erasure Codingappend enginebig data
0 likes · 14 min read

This article explains HBase’s LSM‑Tree compaction fundamentals—including minor and major compaction triggers, file‑selection policies, dynamic throughput throttling, and practical tuning examples that show how adjusting size limits, thread pools, and off‑peak settings can dramatically improve read latency and cluster stability.
HBaseJavaPerformance Tuning
0 likes · 35 min read

Understanding HBase compaction involves knowing its minor and major merge types, trigger mechanisms, file‑selection policies such as RatioBased and Exploring, throttling controls based on file count, and practical tuning of key parameters to avoid latency spikes, as illustrated by real‑world production cases.
HBasebig datacompaction
0 likes · 36 min read

This article details Huolala’s experience building a unified Hive‑to‑HBase pipeline, addressing low development efficiency, lack of monitoring, and HBase instability by evaluating two architectures, implementing a generic Transform tool, optimizing compaction and DistCp, and establishing stability and data‑validation mechanisms.
DistcpHBasebulkload
0 likes · 12 min read

This article explains the LSM‑Tree foundation of OceanBase, details its tiered and leveled compaction strategies, and presents two experiments that observe Mini and Minor compactions under different configuration parameters, revealing how minor freeze and trigger settings affect data movement between L0 and L1 layers.
Database StorageLSM-treeMini Compaction
0 likes · 13 min read

This article introduces the Iceberg table format, explains its core architecture and advantages such as transactionality, implicit partitioning and row‑level updates, details Xiaomi's practical deployments—including CDC pipelines, partition strategies, compaction services, and stream‑batch integration—and outlines future development directions.
FlinkSparkcompaction
0 likes · 20 min read

This article explains how Apache BookKeeper's garbage collection works, detailing minor and major GC triggers, compression strategies, entry log size management, usage calculations, and provides a step‑by‑step code analysis of the GarbageCollectorThread implementation.
Apache BookKeeperEntryLogPulsar
0 likes · 19 min read

This article details Xiaomi's three‑year journey of adopting Apache Doris across dozens of internal services, describing the transition from a Spark‑SQL‑based Lambda architecture to a unified MPP database, performance benchmarks, data ingestion pipelines, compaction tuning, two‑phase commit, single‑replica writes, monitoring, and community contributions.
Apache DorisMPPReal‑Time Analytics
0 likes · 19 min read

This article explains Doris's compaction mechanism, covering its producer‑consumer architecture, tablet scoring, permission control, cumulative and base compaction processes, parameter tuning, monitoring metrics, and manual compaction commands to help optimize performance and resource usage.
Dorisbigdatacompaction
0 likes · 38 min read

This article explains the design and implementation of StarRocks 2.x Primary Key tables, covering real‑time update mechanisms, write and commit workflows, in‑memory primary indexing, compaction, read‑path optimizations, performance benchmarks, and upcoming features such as partial and conditional updates.
IndexingOLAPPrimary Key
0 likes · 19 min read

To meet Cloud Music’s massive algorithm‑feature KV storage needs, the team built RDB—a RocksDB‑based engine within Tair—adding bulk‑load, dual‑version imports, KV‑separation, in‑place sequence appends and protobuf field updates, cutting storage cost, write amplification and latency while scaling to billions of records and millions of QPS.
Algorithm FeaturesKV SeparationKV storage
0 likes · 16 min read

This article explains how Tencent leverages Apache Iceberg together with Flink to build a real‑time data lake pipeline, covering data ingestion, Iceberg's snapshot‑based read/write model, compaction and governance services, Z‑order based query optimization, performance results, and future roadmap.
Apache IcebergFlinkZ-Order
0 likes · 24 min read

This article explains the fundamentals of Log‑Structured Merge (LSM) Trees, covering their write‑ahead log, memtable, SSTable architecture, compaction processes, read/write optimizations, and popular open‑source implementations such as LevelDB, RocksDB, and GoLevelDB.
LSM-treeLevelDBRocksDB
0 likes · 15 min read

LevelDB stores data using an in‑memory Memtable that flushes to immutable tables and disk‑based SSTables, writes are logged then batched and applied through a writer queue, reads check Memtable, immutable Memtable, then SSTables, and background compactions merge tables to improve read performance and reclaim space.
Database InternalsLSM-treeLevelDB
0 likes · 16 min read

This article explains why dropping large amounts of data in a TiDB cluster can trigger compaction flow‑control, leading to write stalls and QPS jitter, and provides step‑by‑step troubleshooting, configuration tweaks, and best‑practice recommendations to resolve the issue.
ConfigurationRegion MergeTiDB
0 likes · 20 min read

This article provides a comprehensive overview of Apache HBase’s architecture, detailing its core components such as RegionServer, HMaster, ZooKeeper, WAL, MemStore, and HFiles, and explains key processes including read/write paths, compaction, region splitting, load balancing, and recovery mechanisms.
Database ArchitectureHBaseNoSQL
0 likes · 17 min read

This article explains HBase compaction—a key operation in the Log‑Structured Merge‑Tree model—covering minor and major compaction differences, trigger conditions, configuration parameters, selection algorithms, thread‑pool handling, and the effects on read and write performance in a big‑data database environment.
HBaseLSMbigdata
0 likes · 10 min read

This article explains how to configure JVM memory, choose appropriate garbage‑collector settings, tune HBase region split policies, optimize BlockCache implementations, and select suitable compaction strategies to improve HBase performance on clusters of various sizes.
BlockCacheDatabase PerformanceHBase
0 likes · 20 min read

This article explains how HBase uses WAL and MemStore to create HFiles, describes the two compaction types (Minor and Major), and provides detailed recommendations for tuning key compaction-related configuration parameters to improve query performance and reduce HDFS impact.
HBaseParameterscompaction
0 likes · 4 min read

This article explains the principles, evolution, modern structures, compaction strategies, optimization techniques such as Bloom filters and partitioning, and concurrency and recovery mechanisms of Log-Structured Merge (LSM) trees, which are widely used in contemporary NoSQL storage systems.
Bloom FilterLSM-treeNoSQL
0 likes · 12 min read

This guide explains how to optimize HBase performance by adjusting JVM memory settings, selecting appropriate garbage collectors, configuring MSLAB and in‑memory compaction, choosing region split policies, tuning BlockCache implementations, and applying suitable compaction policies for different workloads.
BlockCacheHBaseJVM tuning
0 likes · 18 min read

This article describes ByteDance’s extensive improvements to the RocksDB storage engine—including LazyBuffer, Adaptive Map‑based lazy compaction, KV separation, adaptive multi‑index support, extreme compression techniques, and hardware acceleration—to reduce amplification, improve performance, and lower costs for large‑scale database workloads.
CompressionIndexingKV Separation
0 likes · 14 min read

This article explains the core mechanisms of HBase—Flush and Compaction—detailing why they are needed, the conditions that trigger Flush, the types and triggers of Compaction, and provides practical recommendations for tuning the most important configuration parameters to improve write and read performance.
ConfigurationHBaseNoSQL
0 likes · 11 min read

This article explains Cassandra’s row‑oriented storage model, the multi‑step write and read processes, how tombstones and compaction manage data growth, and the impact of its distributed architecture on high availability, fault tolerance, and configurable consistency levels.
CassandraDatabase ArchitectureRead Path
0 likes · 25 min read

This article examines the root causes and performance impact of massive small-file proliferation in traditional data warehouses, explains why HDFS metadata limits scalability, and details how Delta Lake’s custom compaction process can safely merge these files for append-only tables without disrupting reads or writes.
Delta LakeHDFScompaction
0 likes · 5 min read

This article analyzes RocksDB's compaction rate‑limiting source code and presents practical tuning methods—both fixed and auto‑tuned—to mitigate IO spikes in the distributed KV store WTable, improving real‑time read/write latency and stability.
IO optimizationRocksDBcompaction
0 likes · 7 min read

This article provides a comprehensive, illustrated explanation of Apache HBase's architecture, covering its master‑slave components, region management, Zookeeper coordination, data flow for reads and writes, storage structures, compaction processes, fault recovery, and the system's strengths and limitations within the Hadoop ecosystem.
HBaseHadoopNoSQL
0 likes · 21 min read

This article provides an in‑depth technical overview of Alibaba's X‑Engine storage engine, explaining its LSM‑based architecture, compaction optimizations, transaction pipeline, caching strategies, and how these innovations enable low‑cost, high‑throughput OLTP for large‑scale e‑commerce workloads.
LSMPolarDB-Xcompaction
0 likes · 18 min read

This article explains HBase's compaction mechanism, covering why it is needed, the differences between minor and major compaction, the conditions that trigger compaction, key configuration parameters, thread‑pool handling, compaction policies, and how compaction influences read and write performance in a large‑scale NoSQL database.
HBasebigdatacompaction
0 likes · 12 min read

This article presents a detailed post‑mortem of a 10‑node HBase cluster failure caused by excessive region count and memstore pressure, analyzes HDFS and datanode log errors, and outlines configuration adjustments and operational recommendations that restored the service and prevented future outages.
Cluster OutageHBaseHDFS
0 likes · 16 min read

This article presents the design, implementation, and evaluation of X‑Engine, a next‑generation LSM‑Tree based storage engine that offloads compaction to FPGA, achieving up to 50% KV‑interface and 40% SQL‑interface performance gains for write‑intensive OLTP workloads.
FPGALSM-treeX-Engine
0 likes · 19 min read

This article introduces Alibaba’s X‑Engine storage engine, the foundation of the next‑generation distributed database X‑DB, and explains how FPGA‑accelerated compaction and asynchronous scheduling dramatically improve write‑intensive OLTP performance, reduce CPU contention, and achieve up to 50 % throughput gains while maintaining fault tolerance.
FPGALSM-treeX-Engine
0 likes · 21 min read

LevelDB, Google's high-performance key‑value store built on LSM trees, uses an in‑memory skip‑list, immutable memtables, and sstable files organized in multi‑level compaction, offering interfaces for creation, reads, writes, snapshots, and new features like fuzzy search and JSON storage, all explained with diagrams.
Bloom FilterDatabase ArchitectureLSM-tree
0 likes · 11 min read

This article explains how Kafka's LogManager loads existing logs, manages background threads for flushing, checkpointing, cleaning, and compaction, and details the code implementations and strategies for log retention, segment cleanup, and log compression in a distributed storage environment.
KafkaLog Cleaningbackend development
0 likes · 15 min read
