Optimizing HBase‑to‑Hive Data Transfer with SnapshotScanMR to Reduce RegionServer Load

Background: In a business scenario requiring frequent back‑filling of 7‑15‑day data, extracting data offline caused huge pressure on the HBase cluster because ETL jobs had to scan billions of rows.

Old solution: The traditional approach used HBaseStorageHandler to map HBase tables to Hive, then performed ETL extraction into a new Hive table. This generated massive scan requests to HBase region servers, leading to load alerts and resource contention, especially during the nightly peak.

Root cause analysis: HBaseStorageHandler internally invokes the TableScanMR API, which parallelizes scans by splitting them at region boundaries. Each sub‑scan sends a series of next requests to the region server, each returning at most 100 rows or 2 MB. When scanning large tables, the sheer number of next calls overwhelms the region servers, degrading cluster stability and affecting other workloads.

Proposed solution – hbase2hiveBySnapshot: By leveraging HBase’s SnapshotScanMR feature, a new task type was built that first creates a snapshot of the source table, then uses a custom InputFormat to read each HRegion’s HFile directly as a map input. The reduce phase applies user‑defined filters and writes the result to HDFS, which is subsequently loaded into the target Hive table/partition.

Benefits: The snapshot‑based approach bypasses region servers entirely, eliminating their load, cutting task execution time by about 50 %, and achieving roughly a 2× improvement in scanning efficiency. Tests showed no pressure on region servers and improved overall cluster stability.

Further work: Future enhancements may include native filter support, skipping empty HFiles, and more flexible task partitioning (e.g., user‑defined region splits).