How to Compile and Install CDH Hadoop on Kunpeng Cloud: Step‑by‑Step Guide

Today we share the experience of a full‑stack solution provider for industrial IoT that migrated the CDH big‑data component to Kunpeng cloud, describing the problems encountered and the solutions applied.

CDH (Cloudera Distribution including Hadoop) is a Java‑based Hadoop distribution originally built for x86 machines; it must be modified to run on Kunpeng servers.

Environment information

OS: CentOS 7.5 Kernel: 4.14 Software versions: CDH‑hadoop hadoop-2.6.0-cdh5.13.3, Maven 3.5.4, Ant 1.9.4, Git 1.8.3.1, GCC 4.8.5, CMake 3.11.2, JDK 1.8

Dependency installation

yum -y install gcc-c++ autoconf automake libtool cmake svn openssl-devel ncurses-devel

Component installation steps

1. Install JDK 1.8.0

Refer to the Huawei Cloud Kunpeng JDK‑1.8.0 installation guide.

2. Install Maven

wget https://archive.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz

tar -zxvf apache-maven-3.5.4-bin.tar.gz -C /opt/ClouderaHadoopTAR/

Configure environment variables:

vim /etc/profile

export M2_HOME=/opt/ClouderaHadoopTAR/apache-maven-3.5.4

export PATH=$M2_HOME/bin:$PATH

source /etc/profile

mvn -v

3. Install Findbugs

wget http://prdownloads.sourceforge.net/findbugs/findbugs-3.0.1.tar.gz

tar -zxvf findbugs-3.0.1.tar.gz -C /usr/local/hadoopapps/findbugs/

export FINDBUGS_HOME=/usr/local/hadoopapps/findbugs/findbugs-3.0.1/

export PATH=$PATH:$FINDBUGS_HOME/bin

findbugs -version

4. Install Protocol Buffer

yum install protobuf-compiler.aarch64

protoc --version

5. Install Snappy

wget https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz

tar -zxvf snappy-1.1.3.tar.gz

cd snappy-1.1.3/

./configure

make

make install

ls -lh /usr/local/lib | grep snappy

After all dependencies are ready, start compiling CDH.

Compile CDH

Download source package:

wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.13.3.tar.gz

tar -zxvf hadoop-2.6.0-cdh5.13.3.tar.gz

cd hadoop-2.6.0-cdh5.13.3/src

Modify JDK version in pom files (replace 1.7 with 1.8):

sed -i "s/1.7/1.8/g" `grep javaVersion -rl /usr/local/src/hadoop-2.6.0-cdh5.13.3/pom.xml`

sed -i "s/1.7/1.8/g" `grep 1.7 -rl /usr/local/src/hadoop-2.6.0-cdh5.13.3/hadoop-common-project/hadoop-annotations/pom.xml`

Fix malformed HTML tags in source files:

sed -i "s/<\/ul>//g" `grep "</ul>" -l /usr/local/src/hadoop-2.6.0-cdh5.13.3/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java`

Disable DocLint by editing pom.xml and adding inside <properties>:

<additionalparam>-Xdoclint:none</additionalparam>

Ensure protobuf version matches Hadoop (e.g., 2.5.0) and install the matching package.

Compile with Maven:

mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy -Dmaven.javadoc.skip=true

Compilation results are placed in ./hadoop-dist/target/.

Common compilation issues

javah error – set JAVA_HOME correctly.

checksum.cc error – comment out hardware CRC32 implementation lines 582‑584.

Additional optional steps include building libtorrent‑rasterbar.so, wrapping C/C++ code with SWIG for Python, and creating a Python extension module.