Understanding Apache BookKeeper GC: Mechanisms, Triggers, and Code Walkthrough

Apache BookKeeper stores data as log entries in Ledger disks' EntryLog files, and a background GC thread reclaims space by deleting obsolete entries. In practice, old EntryLog files sometimes persist for months, prompting an investigation into the GC mechanism.

BookKeeper GC Overview

BookKeeper isolates write, read, and compaction operations. Multiple Ledgers share a single EntryLog file; the GC thread removes entries that are no longer referenced. If any Ledger in an EntryLog still holds data, the whole file remains on disk.

GC Types and Triggers

Minor GC : Runs by default every hour (configurable via minorCompactionInterval). It triggers only when the useful data ratio in an EntryLog is below minorCompactionThreshold (default 20%). Execution time can be limited with minorCompactionMaxTimeMillis.

Major GC : Runs by default every 24 hours (configurable via majorCompactionInterval) and uses a higher threshold ( majorCompactionThreshold, default 80%). It also respects majorCompactionMaxTimeMillis.

Both thresholds must stay below 100 %, and Minor GC settings must be stricter than Major GC when both are enabled.

Compression Strategies

By Entry Count : Default compresses up to 1 000 entries per run; customizable via compactionRateByEntries.

By Entry Size : Compresses based on total byte size; enable with isThrottleByBytes=true and configure compactionRateByBytes.

Production environments usually prefer size‑based compression because Pulsar batches can produce highly variable entry sizes, which may cause uneven I/O if count‑based compression is used.

GC Trigger Methods

Automatic : The GC thread periodically checks thresholds and runs the appropriate compaction.

Manual : Users can invoke the REST API:

curl -X PUT http://127.0.0.1:8000/api/v1/bookie/gc

with the Bookie's IP and HTTP port (default 8000). GC status can be queried via:

curl http://127.0.0.1:8000/api/v1/bookie/gc_details

Sample JSON response shows flags and counters for major/minor compaction.

Key Code Paths in GarbageCollectorThread

doGcLedgers()

: Compares active Ledgers from RocksDB with metadata stored in ZooKeeper; removes Ledger IDs absent in ZooKeeper using a GarbageCleaner that calls ledgerStorage.deleteLedger. doGcEntryLogs(): Iterates entryLogMetaMap, removes Ledger references that no longer exist in the ledger index, and deletes an EntryLog file if its ledgersMap becomes empty. doCompactEntryLogs(): Sorts EntryLogs by usage (remainingSize/totalSize), scans each file, copies still‑active entries to a new EntryLog via a scanner, flushes the new file, updates RocksDB indexes, and finally removes the old file.

EntryLog Size and Usage Calculation

Each EntryLog defaults to ~1 GB; the limit is defined by MAX_LOG_SIZE_LIMIT = 1 * 1024 * 1024 * 1024. The method reachEntryLogLimit checks if the current file size plus the incoming entry exceeds this limit. Users can adjust the limit via the logSizeLimit configuration.

Usage is computed as remainingSize / totalSize, where totalSize accumulates all entry sizes added to the log and remainingSize decreases when entries are removed during compaction.

Practical Takeaways

Understanding the thresholds helps avoid unnecessary full scans and I/O spikes.

Choosing size‑based compression mitigates uneven I/O caused by large Pulsar batch entries.

Manual GC via REST API provides immediate reclamation when retention policies change.

Reviewing the GC thread code (especially doGcLedgers, doGcEntryLogs, and doCompactEntryLogs) is essential for troubleshooting persistent EntryLog files.

Overall, the GC process balances periodic background reclamation with configurable thresholds to ensure efficient disk usage while minimizing impact on read/write performance.