Understanding HBase Region Auto‑Splitting: Policies, Process, and Pitfalls

Why Region Auto‑Splitting Matters

Region auto‑splitting is a key factor that gives HBase its excellent scalability, and it serves as a model for any distributed system that needs unlimited expansion. Understanding the trigger conditions, split point calculation, availability guarantees, exception handling, and data movement is essential for both users and developers.

Split Policies in HBase 1.2.6

HBase provides six built‑in split policies. The most commonly used are illustrated below:

ConstantSizeRegionSplitPolicy : The default before version 0.94. A split is triggered when the largest store in a region exceeds hbase.hregion.max.filesize. The threshold applies to each store, not the whole region, and the size considered is the compressed file size.

This policy treats large and small tables the same, leading to two problems: a high threshold may prevent small tables from splitting, while a low threshold can cause many tiny regions in large tables.

IncreasingToUpperBoundRegionSplitPolicy : Default from version 0.94 to 2.0. It adjusts the split threshold based on the number of regions on the current region server: (#regions) * (#regions) * (#regions) * flushSize * 2, capped by MaxRegionFileSize. It adapts to both large and small tables but can still generate many small regions in large clusters.

SteppingSplitPolicy : Default from version 2.0. The threshold depends on the region count on the server: if there is only one region, the threshold is flushSize * 2; otherwise it uses MaxRegionFileSize. This reduces the creation of tiny regions for small tables.

Other policies include DisableSplitPolicy (disables splitting) and KeyPrefixRegionSplitPolicy / DelimitedKeyPrefixRegionSplitPolicy, which enforce that keys with the same prefix stay in the same region.

In most cases the default policy is sufficient. It can be set at the column‑family level with the command:

create 'table', {NAME => 'cf', SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy'}

Finding the Split Point

All default policies locate the split point in the middle of the largest store’s largest file, specifically the first row key of the central block. If the calculated row key is the first or last key of the file, no split point is found, and the split is aborted.

A common “no split point” scenario occurs when a file contains only a single block, which often happens in test tables with very few rows.

Split Transaction Workflow

HBase wraps the entire split process in a transaction to guarantee atomicity. The workflow consists of three phases: prepare , execute , and rollback (if needed).

Prepare Phase

Two child HRegionInfo objects are created in memory, containing table name, region name, start key, end key, etc.

A transaction journal is instantiated to record progress for possible rollback.

Execute Phase

The region server updates the ZooKeeper node /region-in-transition to mark the region as SPLITTING.

The master watches this node, updates the region state in memory, and displays the split status in the RIT module.

A temporary .split directory is created under the parent region’s storage path to hold daughter region metadata.

The parent region is closed, flushing all writes to disk; client requests to the parent receive NotServingRegionException.

Reference files are created in the .split folder, each pointing to the parent’s HFile that corresponds to the split point.

Two daughter directories (daughter A and daughter B) are created, and the reference files are populated.

The hbase.meta table is updated to mark the parent region as offline and to record the two daughters.

The daughters are opened and start serving requests.

Rollback Phase

If an exception occurs during the execute phase, HBase uses the journal entries ( JournalEntryType) to clean up partially created files and revert the system to its pre‑split state.

Reference Files

A reference file’s name encodes both the parent region ID and the original HFile name (e.g.,

d24415c4fb44427b8f698143e5c4d9dc.00bb6239169411e4d0ecb6ddfdbacf66

). Its content consists of two parts: the split key and a boolean indicating whether the file represents the upper (true) or lower (false) half of the parent file.

You can view a reference file with Hadoop:

hadoop dfs -cat /hbase-rsgroup/data/default/music/0155388346c3c919d3f05d7188e885e0/cf/d24415c4fb44427b8f698143e5c4d9dc.00bb6239169411e4d0ecb6ddfdbacf66

Data Migration

After the split, the daughter regions contain only metadata (split key, reference files). Actual data from the parent is migrated to the daughters during a major compaction , which reads the parent’s data belonging to each daughter and writes it into the daughter’s HFiles.

Parent Region Cleanup

The HMaster periodically scans the hbase.meta table for regions marked as split. If both daughters no longer have reference files, the parent’s files are eligible for deletion. Deletion is performed only after confirming that no dangling references exist.

Common Production Issues

Long‑running RIT (Region In Transition) states often indicate split failures. The hbck tool can diagnose and fix these problems. Typical commands include: -fixSplitParents – force offline split parents online. -removeParents – offline lingering parents while keeping daughters. -fixReferenceFiles – clean up lingering reference files.

Example error:

ERROR: Found lingering reference file hdfs://mycluster/hbase/news_user_actions/3b3ae24c65fc5094bc2acfebaa7a56de/meta/0f47cda55fa44cf9aa2599079894aed6.b7b3faab86527b88a92f2a248a54d3dc

Such errors may stem from bugs like HBASE‑13331, where the master incorrectly assumes a parent region’s reference files are missing, leading to premature deletion.

Summary

HBase’s region split mechanism is a multi‑step, transactional process that balances scalability with data consistency. By understanding split policies, split point calculation, the prepare‑execute‑rollback workflow, reference file semantics, data migration via compaction, and cleanup logic, operators can both tune performance and troubleshoot common split‑related issues.