Quickly Get Hadoop 2.0 Up and Running: A Minimal Configuration Guide

Overview

Hadoop 2.0 architecture differs from 1.0; configuration files are in a new directory and YARN must be configured. This guide provides the simplest configuration to get Hadoop 2.0 running quickly.

Prerequisites

Version: Hadoop‑2.2.0 (first stable Hadoop 2.0 release, 15 Oct 2013).

Two machines are used: one master (hadoop2-m1) and one slave (hadoop2-s1).

Installation Directory

Set HADOOP_HOME=/your/path/to/hadoop-2.2.0 and HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop.

Configuration Files

Edit the four XML files under $HADOOP_CONF_DIR:

core-site.xml

hdfs-site.xml

mapred-site.xml

yarn-site.xml

Set JAVA_HOME

export JAVA_HOME=/your/path/to/jdkdir

core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop2-m1:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/tmp/hadoop2.0</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/home/dfs/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/home/dfs/data</value>
  </property>
  <property>
    <name>dfs.permissions</name>
    <value>false</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

yarn-site.xml

<configuration>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>hadoop2-m1:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>hadoop2-m1:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>hadoop2-m1:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>hadoop2-m1:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>hadoop2-m1:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

Start Services

Format the namenode:

cd $HADOOP_HOME
bin/hdfs namenode -format

On the master start namenode and ResourceManager:

sbin/hadoop-daemon.sh start namenode
sbin/yarn-daemon.sh start resourcemanager

On the slave start datanode and NodeManager:

sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start nodemanager

Start proxy server and history server on the master:

sbin/yarn-daemon.sh start proxyserver
sbin/mr-jobhistory-daemon.sh start historyserver

Verification

Check the web UI at http://hadoop2-m1:50070/dfshealth.jsp and http://hadoop2-m1:8088/cluster/nodes.

Run HDFS commands to create a test directory and copy files, then run a sample MapReduce job:

bin/hdfs dfs -mkdir /test/input1
bin/hdfs dfs -put NOTICE.txt /test/input1/
bin/hdfs dfs -put README.txt /test/input1/
bin/hdfs dfs -cat /test/input1/NOTICE.txt
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar grep /test/input1 /test/output1 'code'

Tips

Perform the minimal configuration first to ensure Hadoop 2.0 starts correctly before adding advanced features such as HA or federation.

Reference

Apache Hadoop documentation:

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/ClusterSetup.html