HBase Application and High‑Availability Practices

Ping An Technology’s HBase deployment spans hundreds of physical machines with petabyte‑scale data, serving insurance, security, finance, and AI workloads. The cluster size and application scenarios are illustrated with diagrams.

1. Current Usage – Two aspects are highlighted: cluster scale (hundreds of nodes, PB‑level data) and diverse application domains such as property insurance, life insurance, information security, unified accounts, intelligent engine, and credit services.

2. Problems Solved for Users – Massive data storage, performance, reliability, and migration challenges are addressed. HBase’s online horizontal scaling accommodates explosive data growth, and migration from traditional databases simplifies maintenance.

Client‑Side Optimizations – Reduce IPC calls by batching Scan requests, use single‑row Get or batch Get via a list, and limit returned columns to improve latency. Disable caching for write‑heavy workloads to avoid hot‑data eviction.

Server‑Side Optimizations – Balance operations via balance_switch (true for auto‑balance, false to disable) or manual balancer. Monitor region distribution using the ratio of total regions to regionservers, keeping it between 0.8 and 1.2 to trigger safe rebalancing.

Additional server tweaks include enabling Block Cache for read efficiency, adjusting external memory to improve cache hit rate and GC behavior, and configuring Compaction to maintain data locality (target level 1) during low‑traffic windows.

3. High Availability – HBase provides master failover and region replication to avoid single‑point failures. Data migration scenarios cover cluster‑to‑cluster moves (using hadoop distcp) and Hive‑to‑HBase transfers via HFile bulk‑load or API‑driven bulk load.

hadoop distcp -Dmapreduce.job.queuename=queue_name-update --skipcrccheck -m 120 hdfs://ip:port/hbase_path_old/tabname/hbase_path_new/tabname

Migration prerequisites include enabling YARN, ensuring firewall ports (NameNode, DataNode) are reachable, and handling lease recovery with hdfs debug recoverLease -path [hdfs_file_path] when files are still open.

4. Monitoring and Repair – Collect all HBase metrics (e.g., via http://regionserver-ip:port/jmx) to assess health. Visualize core metrics to detect cluster jitter or latency spikes. Automated repair policies address process availability, load imbalance, and RIT (Region In Transition) issues, with manual interventions for bugs such as HBASE‑17682.

5. Future Plans – Lower HBase entry barriers, enhance intelligent monitoring and self‑repair, and conduct a POC for HBase 2 with version upgrades.