How to Build a Beginner Hadoop Cluster on CentOS 7

Apache Hadoop

Apache Hadoop is an open‑source framework released under the Apache 2.0 license that enables data‑intensive distributed applications on commodity hardware. It implements the MapReduce programming model and a distributed file system (HDFS) based on Google’s MapReduce and GFS papers, providing high bandwidth and automatic handling of node failures.

Hadoop 2.0 Ecosystem

The Hadoop 2.0 release adds a resource‑management layer (YARN) and a collection of complementary projects that together form a full data‑processing stack.

HDFS

HDFS (Hadoop Distributed File System) gives every server in a cluster direct access to data. It replicates blocks across multiple nodes, so a node failure does not stop computation and data remains redundant. HDFS stores data in its native format without requiring a predefined schema, allowing both structured and unstructured inputs.

MapReduce

MapReduce is a parallel‑processing model that splits a large data set into independent tasks that run on many nodes. Programmers can write Java APIs without needing expertise in distributed parallel programming; the framework handles task distribution, fault tolerance, and result aggregation.

ZooKeeper

ZooKeeper provides coordination services for distributed applications. It maintains configuration, naming, synchronization, and group membership in memory with replication. An ensemble consists of a leader (handling all write operations) and followers (serving reads); the ensemble remains available even if some nodes fail.

HBase

HBase is a scalable, high‑reliability, column‑oriented NoSQL database built on top of HDFS. It follows Google’s BigTable model, storing data as a sparse, sorted map of row key, column key, and timestamp. HBase offers random, real‑time read/write access to massive data sets, and stored data can be processed with MapReduce, tightly coupling storage and computation.

Hive

Hive is a data‑warehouse tool that maps structured files to database‑like tables and provides an SQL‑like interface (HiveQL). Queries are translated into MapReduce jobs, enabling offline batch processing. Hive supports scalability, user‑defined functions, fault tolerance, flexible input formats (e.g., txt, rc, sequence), and partitioned tables (e.g., by date).

Pig

Pig is a high‑level scripting language that simplifies common Hadoop tasks. Scripts are automatically compiled into optimized MapReduce jobs. Pig supports custom data types via Java extensions and lets users focus on business logic rather than low‑level MapReduce programming.

Mahout

Mahout provides scalable implementations of classic machine‑learning algorithms—including clustering, classification, recommendation, and frequent‑itemset mining—as well as I/O tools and integrations with external storage systems such as relational databases, MongoDB, and Cassandra.

Sqoop

Sqoop is a bulk‑transfer tool for moving data between Hadoop and structured data stores. It can import data from databases (MySQL, Oracle, SQL Server, etc.) into HDFS, Hive, or HBase, and export processed data back to those systems, typically via simple command‑line scripts.

Flume

Flume (provided by Cloudera) is a highly available, reliable, distributed system for collecting, aggregating, and transporting massive log data. It supports custom sources, simple processing (filtering, format conversion), and flexible sinks, making it suitable for complex logging environments.

Chukwa

Chukwa is an open‑source monitoring system built on HDFS and MapReduce. It provides a flexible toolkit for collecting, visualizing, and analyzing monitoring data from large distributed environments.

Oozie

Oozie is a workflow engine that runs Hadoop jobs (MapReduce, Pig, Hive, Sqoop, etc.). Workflows are defined in XML (HPDL) and submitted via HTTP. Dependencies between jobs are expressed in the workflow, enabling scheduled and conditional execution.

Ambari

Ambari is a web‑based tool for provisioning, managing, and monitoring Hadoop clusters. It centralizes management of components such as HDFS, MapReduce, Hive, HBase, ZooKeeper, Oozie, Pig, and Sqoop.

YARN

YARN (Yet Another Resource Negotiator), introduced in Hadoop 2.0, adds a resource‑management and scheduling layer. It enables multiple frameworks—including MapReduce, Hive, HBase, Pig, and Spark—to share cluster resources uniformly. YARN follows a master‑slave architecture: the ResourceManager (master) performs global resource allocation, while NodeManagers (slaves) manage resources and tasks on each DataNode.

For a step‑by‑step installation guide, see the referenced blog post.

https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/details/119335883