HBase Data Migration from Version 0.94.15 to 1.2.1: Issues and Solutions

This article details the HBase data migration process from a customer cluster (Version 0.94.15) to Tencent's big data suite cluster (Version 1.2.1). The migration involved 500GB of HBase data and 5TB of Solr data.

Issue 1: Hardware Clock Synchronization - HBase exhibited intermittent failures with component stoppage. Analysis revealed significant hardware clock drift. The issue was resolved by running hwclock -w to synchronize the hardware clock. The initialization script only synchronized hardware time for Tencent Cloud environment machines, which has since been optimized.

Issue 2: Hostname and DNS Resolution - HBase experienced component failures again. The error log showed:

ERROR [regionserver//10.0.0.106:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=10.0.0.106, but now=host-10-0-0-106.openstacklocal

. The issue was traced to the search openstacklocal entry in /etc/resolv.conf caused by abnormal VM behavior. After commenting out the search entry and stopping the nscd service, HBase ran normally after restart.

Issue 3: Snappy Compression Support - During table migration using import/export tools, the error occurred:

org.apache.apache.hadoop.HBase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test

. The hadoop checknative command showed snappy was not supported despite snappy RPM being installed. The solution involved compiling Hadoop native libraries with snappy support on tlinux1.2, replacing existing native libraries in hadoop/lib, and adding HADOOP_HOME in HBase-env.sh. After full HBase restart, snappy support was confirmed.

Migration Methods - Two approaches were tested:

1) Brute Force Migration : Direct HDFS directory transfer from source to target cluster. Key steps include: locating source table's HDFS directory, moving to target cluster's table root directory, manually creating ./tabledesc directory and moving .tableinfo file (since version 0.94.15 stores it in root while 1.2.1 uses ./tabledesc), modifying file ownership, restarting HBase components, then running HBase hbck -fixMeta and HBase hbck -fixAssignments to repair metadata and assignments.

2) Import/Export Method : Requires manually creating target table in destination cluster. However, the ENCODE_ON_ON_DISK parameter is deprecated in HBase 1.2.1 (moved to a different field structure), causing import failures. The brute force method successfully preserved this parameter.

Conclusion : The brute force migration method proved more reliable for cross-version migration, especially when dealing with deprecated parameters like ENCODE_ON_DISK.