Understanding Doris Compaction Mechanism and Optimization Strategies

Doris stores data using an LSM‑Tree‑like structure where writes are appended, requiring background compaction to merge multiple data versions and improve read efficiency. Compaction reduces data fragmentation, eliminates delete markers, and increases aggregation density.

The compaction framework follows a producer‑consumer model: a single producer thread continuously creates compaction tasks and submits them to a configurable thread pool for execution, as illustrated in the architecture diagram.

Tablets are selected based on a configurable score: tablet_score = k1 * scan_frequency + k2 * compaction_score, where k1 and k2 are set via compaction_tablet_scan_frequency_factor and compaction_tablet_compaction_score_factor. The highest‑scoring tablets are chosen for compaction.

To limit memory consumption, Doris introduces a permission mechanism. Each compaction task must acquire a number of permits proportional to the total segment files it will merge. Permits are defined by total_permits_for_compaction_score; tasks wait if insufficient permits are available.

Compaction tasks consist of two stages: preparation (selecting eligible rowsets) and execution (merging the selected rowsets). The preparation stage filters tablets based on criteria such as disk location, ongoing operations, failure intervals, and then picks the tablet with the highest score.

Cumulative compaction merges recent rowsets after a cumulative point (CP). It follows five steps: calculate CP, generate candidate rowsets, select input rowsets, execute the merge, and update CP. Two policies exist: size_based (default) and num_based . Promotion size is computed as promotion_size = base_rowset_size * ratio (ratio configurable via cumulative_size_based_promotion_ratio) and bounded by cumulative_size_based_promotion_size_mbytes and cumulative_size_based_promotion_min_size_mbytes.

Base compaction merges all rowsets before the CP. It checks version continuity, overlapping segments, rowset count, size ratio, and time interval (controlled by base_compaction_num_cumulative_deltas, base_compaction_interval_seconds_since_last_operation, etc.) before proceeding.

Optimization strategies include increasing max_compaction_threads or compaction_task_num_per_disk to allow more parallel tasks, adjusting promotion thresholds ( cumulative_size_based_promotion_ratio, cumulative_size_based_promotion_size_mbytes), and tuning limits such as max_cumulative_compaction_num_singleton_deltas to control task size. Disabling auto‑compaction via disable_auto_compaction is useful for debugging.

Monitoring can be done through BE metrics (e.g., BE_BASE_COMPACTION_SCORE, BE_CUMU_COMPACTION_SCORE) in Prometheus/Grafana or via ADMIN SHOW BACKEND METRIC commands. Memory usage is visible at http://be_host:webserver_port/mem_tracker and disk I/O via BE_DISK_IO. Manual compaction can be triggered with HTTP APIs, for example:

curl -X POST http://192.168.1.1:8040/api/compaction/run?tablet_id=106818600&schema_hash=6979334&compact_type=cumulative

By understanding these mechanisms and tuning the relevant parameters, operators can ensure that compaction keeps pace with data ingestion, minimizes write amplification, and maintains query performance.