OpenAI shows that enabling retained reasoning and context compaction boosts ARC‑AGI‑3 performance from 13.3% to 38.3% (about three‑fold) by preserving the agent’s working memory, a finding that reshapes how we evaluate and design AI agents beyond raw model weights.
A Milvus delete command looks simple, but the primary key is first stored in an L0 delta segment, then triggers compaction that merges the delete mark into L1/L2 segments, passes through four distinct segment states, and involves multiple seal strategies, rollback fields, and version‑specific behaviours, illustrating why deletion is the system's most intricate operation.
When Milvus appears to keep disk usage unchanged after deleting a million vectors, the cause is not a bug but a sophisticated compaction system that splits the single compact() API into eleven enum values, six independent policies, and seven special handling paths that together manage different kinds of data waste and ensure safe, incremental reclamation.
The article presents a comprehensive architecture for production‑grade autonomous agents, detailing failure modes, four distinct memory types, a nightly observation engine that turns patterns into procedural rules, tier‑aware decay scoring, context budgeting, GDPR‑compliant deletion, and a step‑by‑step maintenance pipeline.
This article enumerates ten uncommon, compaction‑related problems encountered in large‑scale Paimon deployments, explains their root causes—such as RPC timeouts, snapshot expiration, file corruption, and write conflicts—and provides concrete configuration tweaks and operational steps to resolve each issue.
The article explains how Anthropic engineers treat the limited token budget of large language models as a finite resource, detailing static configuration, runtime retrieval, and long‑task strategies such as compaction, structured notes, and sub‑agent architectures to build reliable, efficient AI agents.
The article explains Claude Code's multi‑level fault‑tolerance for max_output_tokens errors, detailing dynamic token allocation, automatic model downgrade, environment‑variable controls, StopFailure hooks, and their coordination with compaction to prevent agents from getting stuck during long‑running tasks.
The article analyzes how five Agent Harness frameworks—Pi, OpenClaw, Claude Code, Letta, and Alyx—handle context windows, file pagination, tool result limits, session pruning, and sub‑agent isolation, revealing convergent design patterns that treat the context as a managed memory system.
This article analyzes the DisCoGC algorithm introduced by ByteDance, explaining how its discard‑centric garbage collection eliminates the write‑amplification vs. space‑amplification trade‑off in log‑structured storage, details the engineering challenges of multi‑layer deployment, and presents production results showing up to 20% TCO reduction without impacting latency.
This article analyzes Anthropic's Claude Code session‑management features, explaining how context rot limits effective token usage, what the 1 M‑token window actually stores, and when to use the five built‑in actions—Continue, /rewind, /clear, Compact and Subagent—to keep long‑running AI tasks reliable and efficient.
The article explains how organizing the prompt, tool schemas, dialogue history, and retrieved documents—collectively the context window—affects an AI agent’s decisions, introduces the concepts of Lost‑in‑the‑Middle, Thinking Tokens, tool‑response caching, compaction versus SubAgent strategies, and shows a step‑by‑step evolution that raised accuracy from 60 % to over 95 %.
The article provides a detailed technical walkthrough of OpenClaw's three‑stage context‑management framework—including pre‑emptive pruning, LLM‑driven compaction, and overflow‑recovery truncation—showing how each layer protects long‑running AI agent sessions from exceeding token windows while preserving essential information.
The article explains why writing code alone does not guarantee delivery, outlines three core challenges for long‑running agents—process reuse, execution, and context continuity—and presents a practical framework of Skills, Shell, and Compaction together with ten actionable recommendations, security guidelines, and implementation steps for teams.
The article explains OpenAI’s new agent primitives—Skills, a hosted Shell environment, and server‑side Compaction—detailing how they enable long‑running, reliable AI agents, provides practical design patterns and tips, and compares this approach with the open‑source OpenClaw framework.
This article explains how StarRocks mitigates the hidden risks of massive one‑time data imports in a storage‑compute separated architecture by redesigning the write path to spill to local disk, merge centrally, and write to object storage, resulting in fewer small files, higher write throughput, and more stable query performance.
When a Kubernetes cluster’s etcd reaches its default 2 GB quota, it triggers a “NOSPACE” alarm that blocks all write operations, causing critical services to fail; this guide explains the root cause, how to diagnose the issue with etcdctl, and step‑by‑step remediation including compaction, defragmentation, and quota expansion.
The article walks through a complete Zig implementation of an LSM‑Tree, detailing its in‑memory skip‑list MemTable, immutable SSTable blocks with compression and Bloom filters, write‑ahead logging, iterator hierarchy for reads, and multi‑level compaction logic that merges and rewrites SSTables.
This presentation describes Tencent's real-time lakehouse architecture, including data lake compute, management, and storage layers, and details the intelligent optimization services—such as compaction, indexing, clustering, and auto-engine—designed to improve query performance, storage cost, and operational efficiency for large-scale data processing.
Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.
The article explains Paimon's changelog‑producer capability, detailing its purpose, storage format, and the four generation modes—None, Input, Lookup, and Full Compaction—while comparing their costs, implementation details, and suitability for different data sources such as CDC.
This article explains the characteristics of Paimon primary key tables, covering bucket selection, cross‑partition update issues, recommended record‑level expiration settings, and two approaches to handle file compaction, including configuration tweaks and dedicated compaction tasks.
The article explains how Linux manages memory page watermarks, when the allocator falls back to kswapd, and the exact conditions that trigger direct compaction via __alloc_pages_direct_compact, then walks through the core compaction functions—try_to_compact_pages, compact_zone_order, compact_zone, and the page‑migration helpers—illustrated with flow diagrams and real kernel code.
This article details a step‑by‑step investigation of a compaction‑stuck problem in OceanBase, covering background, environment setup, view and log analysis, root‑cause identification related to clock drift, and the corrective actions taken to restore normal merging.
This article dissects the Linux kernel's page allocation mechanisms, explaining how alloc_pages() follows a fast‑path using low watermarks and falls back to a slow‑path that triggers kswapd, direct reclaim, and compaction, while also detailing the corresponding page‑freeing functions and their internal data structures.
Tencent’s real‑time lakehouse combines Spark, Flink, StarRocks and Presto compute layers with Iceberg‑based management and HDFS/COS storage, and its Intelligent Optimize Service—comprising Compaction, Expiration, Cleaning, Clustering, Index and Auto‑Engine modules—automatically reduces merge time, improves query performance, enables secondary indexing, and dynamically routes hot partitions, while future plans target cold/hot separation, materialized view acceleration, and AI‑driven optimizations.
This article introduces Apache Hudi's table services, explaining the concepts, execution modes, and detailed workflows of compaction, cleaning, and indexing, and how they optimize storage layout and read/write performance in large‑scale data lake environments.
This article outlines practical production‑level optimizations for Apache Doris compaction, covering vertical, segment, and single‑replica compaction methods, compaction policies, concurrency controls, and data‑ingestion tuning to improve import speed and query performance in OLAP workloads.
This article provides a comprehensive overview of Apache Doris, explaining its frontend and backend nodes, storage structures such as tablets, rowsets, and segments, replication mechanisms, partitioning versus bucketing, indexing types, compaction processes, and high‑availability strategies through a detailed Q&A format.
This article explains Paimon's core concepts—including snapshots, partitions, buckets, consistency guarantees, file layout, LSM‑tree organization, and compaction strategies—while also covering table management tasks such as snapshot expiration, rollback, partition expiration, and small‑file mitigation techniques.
The article explains the Log-Structured Merge-Tree (LSM) architecture, its core components, advantages and disadvantages, and demonstrates how it is employed in big‑data platforms such as HBase and Apache Druid to achieve high‑throughput writes and scalable query processing.
This article explains StarRocks' compaction process that merges multiple data versions into larger files to reduce I/O, details the scheduler and executor roles, shows how to monitor and control compaction via SQL commands, and provides tuning parameters and best‑practice recommendations for optimal performance.
Apache Paimon provides multiple table modes—including primary key tables with fixed or dynamic buckets, Append scalable and queue tables—each with specific configurations, compaction behavior, and suitable scenarios, and the article explains their structures, performance considerations, and how to use them with Flink.
This article explains the Log-Structured Merge Tree (LSM-Tree) architecture, describing its immutable storage design, the roles of WAL, MemTable, ImmuTable, and SSTable, and detailing the write workflow, compaction process, and the associated read, space, and write amplification challenges.
This article explains LSM‑Tree based HBase compaction concepts, compares Minor and Major compactions, and shares practical tuning steps—including disabling automatic major compactions, controlling merge size, leveraging off‑peak windows, and improving merge efficiency—to reduce I/O, CPU usage, and latency in production environments.
This article explains the Linux page‑reclaim mechanism, its goals, common techniques, the allocation paths, LRU data structures, and provides an in‑depth walkthrough of the kernel source code for slow‑path reclaim, direct reclaim, and memory compaction, including all relevant functions and code snippets.
This article explains RocketMQ's random indexing mechanism, detailing its on‑disk three‑segment hash table structure, the compact format conversion process, multi‑threaded write and query workflows, layered system design, crash‑recovery strategy, and comparisons with RocksDB and InnoDB storage engines.
This article explains how to parse OceanBase logs to trace the tenant freeze dump workflow, detailing the roles and log sequences of the freeze check thread, LSFreeze, Flush, DagScheduler, and MiniMerge threads, and illustrating each step with actual log excerpts and code snippets.
This article presents Baidu Cloud's block storage architecture, comparing replication and erasure‑coding fault‑tolerance methods, detailing the challenges of applying EC to mutable block data, and describing a two‑layer append‑engine solution with selective 3‑replica caching, cost‑benefit compaction, and performance optimizations for low‑cost, high‑throughput storage.
This article explains HBase’s LSM‑Tree compaction fundamentals—including minor and major compaction triggers, file‑selection policies, dynamic throughput throttling, and practical tuning examples that show how adjusting size limits, thread pools, and off‑peak settings can dramatically improve read latency and cluster stability.
Understanding HBase compaction involves knowing its minor and major merge types, trigger mechanisms, file‑selection policies such as RatioBased and Exploring, throttling controls based on file count, and practical tuning of key parameters to avoid latency spikes, as illustrated by real‑world production cases.
This article details Huolala’s experience building a unified Hive‑to‑HBase pipeline, addressing low development efficiency, lack of monitoring, and HBase instability by evaluating two architectures, implementing a generic Transform tool, optimizing compaction and DistCp, and establishing stability and data‑validation mechanisms.
This article explains the LSM‑Tree foundation of OceanBase, details its tiered and leveled compaction strategies, and presents two experiments that observe Mini and Minor compactions under different configuration parameters, revealing how minor freeze and trigger settings affect data movement between L0 and L1 layers.
This article introduces the Iceberg table format, explains its core architecture and advantages such as transactionality, implicit partitioning and row‑level updates, details Xiaomi's practical deployments—including CDC pipelines, partition strategies, compaction services, and stream‑batch integration—and outlines future development directions.
This article explains how Apache BookKeeper's garbage collection works, detailing minor and major GC triggers, compression strategies, entry log size management, usage calculations, and provides a step‑by‑step code analysis of the GarbageCollectorThread implementation.
This article details Xiaomi's three‑year journey of adopting Apache Doris across dozens of internal services, describing the transition from a Spark‑SQL‑based Lambda architecture to a unified MPP database, performance benchmarks, data ingestion pipelines, compaction tuning, two‑phase commit, single‑replica writes, monitoring, and community contributions.
This article explains Doris's compaction mechanism, covering its producer‑consumer architecture, tablet scoring, permission control, cumulative and base compaction processes, parameter tuning, monitoring metrics, and manual compaction commands to help optimize performance and resource usage.
This article explains the design and implementation of StarRocks 2.x Primary Key tables, covering real‑time update mechanisms, write and commit workflows, in‑memory primary indexing, compaction, read‑path optimizations, performance benchmarks, and upcoming features such as partial and conditional updates.
To meet Cloud Music’s massive algorithm‑feature KV storage needs, the team built RDB—a RocksDB‑based engine within Tair—adding bulk‑load, dual‑version imports, KV‑separation, in‑place sequence appends and protobuf field updates, cutting storage cost, write amplification and latency while scaling to billions of records and millions of QPS.
This article explains how Tencent leverages Apache Iceberg together with Flink to build a real‑time data lake pipeline, covering data ingestion, Iceberg's snapshot‑based read/write model, compaction and governance services, Z‑order based query optimization, performance results, and future roadmap.
This article explains the fundamentals of Log‑Structured Merge (LSM) Trees, covering their write‑ahead log, memtable, SSTable architecture, compaction processes, read/write optimizations, and popular open‑source implementations such as LevelDB, RocksDB, and GoLevelDB.
LevelDB stores data using an in‑memory Memtable that flushes to immutable tables and disk‑based SSTables, writes are logged then batched and applied through a writer queue, reads check Memtable, immutable Memtable, then SSTables, and background compactions merge tables to improve read performance and reclaim space.
This article explains why dropping large amounts of data in a TiDB cluster can trigger compaction flow‑control, leading to write stalls and QPS jitter, and provides step‑by‑step troubleshooting, configuration tweaks, and best‑practice recommendations to resolve the issue.
This article provides a comprehensive overview of Apache HBase’s architecture, detailing its core components such as RegionServer, HMaster, ZooKeeper, WAL, MemStore, and HFiles, and explains key processes including read/write paths, compaction, region splitting, load balancing, and recovery mechanisms.
This article explains HBase compaction—a key operation in the Log‑Structured Merge‑Tree model—covering minor and major compaction differences, trigger conditions, configuration parameters, selection algorithms, thread‑pool handling, and the effects on read and write performance in a big‑data database environment.
This article explains how to configure JVM memory, choose appropriate garbage‑collector settings, tune HBase region split policies, optimize BlockCache implementations, and select suitable compaction strategies to improve HBase performance on clusters of various sizes.
This article explains how HBase uses WAL and MemStore to create HFiles, describes the two compaction types (Minor and Major), and provides detailed recommendations for tuning key compaction-related configuration parameters to improve query performance and reduce HDFS impact.
This article explains the principles, evolution, modern structures, compaction strategies, optimization techniques such as Bloom filters and partitioning, and concurrency and recovery mechanisms of Log-Structured Merge (LSM) trees, which are widely used in contemporary NoSQL storage systems.
This guide explains how to optimize HBase performance by adjusting JVM memory settings, selecting appropriate garbage collectors, configuring MSLAB and in‑memory compaction, choosing region split policies, tuning BlockCache implementations, and applying suitable compaction policies for different workloads.
This article describes ByteDance’s extensive improvements to the RocksDB storage engine—including LazyBuffer, Adaptive Map‑based lazy compaction, KV separation, adaptive multi‑index support, extreme compression techniques, and hardware acceleration—to reduce amplification, improve performance, and lower costs for large‑scale database workloads.
This article explains the core mechanisms of HBase—Flush and Compaction—detailing why they are needed, the conditions that trigger Flush, the types and triggers of Compaction, and provides practical recommendations for tuning the most important configuration parameters to improve write and read performance.
This article explains Cassandra’s row‑oriented storage model, the multi‑step write and read processes, how tombstones and compaction manage data growth, and the impact of its distributed architecture on high availability, fault tolerance, and configurable consistency levels.
This article examines the root causes and performance impact of massive small-file proliferation in traditional data warehouses, explains why HDFS metadata limits scalability, and details how Delta Lake’s custom compaction process can safely merge these files for append-only tables without disrupting reads or writes.
This article analyzes RocksDB's compaction rate‑limiting source code and presents practical tuning methods—both fixed and auto‑tuned—to mitigate IO spikes in the distributed KV store WTable, improving real‑time read/write latency and stability.
This article provides a comprehensive, illustrated explanation of Apache HBase's architecture, covering its master‑slave components, region management, Zookeeper coordination, data flow for reads and writes, storage structures, compaction processes, fault recovery, and the system's strengths and limitations within the Hadoop ecosystem.
This article provides an in‑depth technical overview of Alibaba's X‑Engine storage engine, explaining its LSM‑based architecture, compaction optimizations, transaction pipeline, caching strategies, and how these innovations enable low‑cost, high‑throughput OLTP for large‑scale e‑commerce workloads.
This article explains HBase's compaction mechanism, covering why it is needed, the differences between minor and major compaction, the conditions that trigger compaction, key configuration parameters, thread‑pool handling, compaction policies, and how compaction influences read and write performance in a large‑scale NoSQL database.
This article presents a detailed post‑mortem of a 10‑node HBase cluster failure caused by excessive region count and memstore pressure, analyzes HDFS and datanode log errors, and outlines configuration adjustments and operational recommendations that restored the service and prevented future outages.
This article presents the design, implementation, and evaluation of X‑Engine, a next‑generation LSM‑Tree based storage engine that offloads compaction to FPGA, achieving up to 50% KV‑interface and 40% SQL‑interface performance gains for write‑intensive OLTP workloads.
This article introduces Alibaba’s X‑Engine storage engine, the foundation of the next‑generation distributed database X‑DB, and explains how FPGA‑accelerated compaction and asynchronous scheduling dramatically improve write‑intensive OLTP performance, reduce CPU contention, and achieve up to 50 % throughput gains while maintaining fault tolerance.
LevelDB, Google's high-performance key‑value store built on LSM trees, uses an in‑memory skip‑list, immutable memtables, and sstable files organized in multi‑level compaction, offering interfaces for creation, reads, writes, snapshots, and new features like fuzzy search and JSON storage, all explained with diagrams.
This article explains how Kafka's LogManager loads existing logs, manages background threads for flushing, checkpointing, cleaning, and compaction, and details the code implementations and strategies for log retention, segment cleanup, and log compression in a distributed storage environment.
