HBase Optimization: JVM Tuning, Region Split Policies, BlockCache, and Compaction Strategies

HBase clusters typically allocate only 1 GB of memory to Master and RegionServer processes, which is insufficient; the MemStore consumes about 0.4 GB by default. Increasing JVM heap sizes, for example using export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xms2g -Xmx2g" and

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xms8g -Xmx8g"

, can greatly improve performance, while leaving at least 10 % of RAM for the OS.

When allocating memory on a 16 GB node that runs MapReduce, RegionServer, and DataNode, a common split is 2 GB for the OS, 8 GB for MapReduce, 4 GB for HBase RegionServer, 1 GB for TaskTracker, and 1 GB for DataNode; adjustments are needed if MapReduce is not present.

For garbage‑collection tuning, HBase primarily runs on RegionServer, so Full GC settings matter most. The JVM offers four collectors: SerialGC, ParallelGC (default for young generations in JDK 8), CMS (for old generations), and G1GC (optimised for >32 GB memory). Typical configurations are:

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xms8g -Xmx8g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC"

(ParallelGC + CMS) or

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xms8g -Xmx8g -XX:+UseG1GC -XX:MaxGCPauseMillis=100"

(G1GC). Choose G1GC only for large (32–64 GB) nodes; otherwise use the ParallelGC+CMS combo and enable detailed GC logging with

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintAdaptiveSizePolicy

.

HBase 2.x introduces MSLAB (Memstore‑Local Allocation Buffers) to manage MemStore memory more efficiently. Relevant parameters include hbase.hregion.memstore.mslab.enabled (default true), hbase.hregion.memstore.mslab.chunksize (2 MB), hbase.hregion.memstore.mslab.max.allocation (256 KB), and chunk‑pool size settings.

Region splitting can be automatic or manual. Automatic policies include:

ConstantSizeRegionSplitPolicy – splits when a region exceeds hbase.hregion.max.filesize.

IncreasingToUpperBoundRegionSplitPolicy (default) – uses

Math.min(tableRegionCount^3 * initialSize, defaultRegionMaxFileSize)

to compute a dynamic limit.

KeyPrefixRegionSplitPolicy – ensures rows with the same prefix stay in the same region (parameter KeyPrefixRegionSplitPolicy.prefix_length).

DelimitedKeyPrefixRegionSplitPolicy – similar to KeyPrefix but uses a delimiter (e.g., ‘_’) to define the prefix.

BusyRegionSplitPolicy – splits hot regions to alleviate hotspots.

DisabledRegionSplitPolicy – disables automatic splitting.

Recommended practice is to pre‑split tables before loading data and then keep automatic splitting enabled to handle hotspot formation.

BlockCache is a per‑RegionServer cache that speeds reads by storing frequently accessed blocks. It can be disabled per column family with

alter 'testTable', CONFIGURATION=>{NAME=>'cf',BLOCKCACHE=>'false'}

. Implementations include LRUBlockCache, SlabCache (now deprecated), and BucketCache. BucketCache can use heap, off‑heap, or file storage (default off‑heap) and supports multiple bucket sizes defined by hbase.bucketcache.bucket.sizes. Configuration keys such as hbase.bucketcache.ioengine, hbase.bucketcache.size, and -XX:MaxDirectMemorySize control its behaviour.

Compaction merges HFiles to reclaim space and improve read performance. Minor Compaction merges a few files frequently, removing expired TTL data; Major Compaction merges all files, also deleting manually removed cells and old versions, typically scheduled every 7 days.

Compaction policies include:

RatioBasedCompactionPolicy – now superseded due to aggressive merging.

ExploringCompactionPolicy – default since HBase 0.96, selects files based on size ratios and configurable thresholds.

FIFOCompactionPolicy – deletes only fully expired files; unsuitable when TTL is absent or MIN_VERSIONS > 0.

DateTieredCompactionPolicy – groups files by age windows (default 6 h) and is ideal for workloads that read recent data.

StripeCompactionPolicy – splits large regions into stripes for stable reads; best for regions >2 GB with uniformly distributed rowkeys.

Choosing the right policy depends on data characteristics: use DateTiered for time‑ordered, frequently accessed recent data; Stripe for large, evenly distributed tables without heavy deletions; FIFO only for very short‑lived data.