Introduction to Hadoop: Architecture, HDFS, MapReduce, and Common Commands

With the growing big‑data needs of various product lines, QA engineers are increasingly required to understand large‑scale data processing techniques such as Hadoop, which is introduced in this article.

Hadoop is an Apache open‑source project inspired by Google File System (GFS) and the MapReduce paper; its two core components are HDFS, a distributed file system that stores massive files in 64 MB blocks with replication, and MapReduce, a programming model that provides distributed computation over those files.

Hadoop consists of a cluster side and a client side. The cluster contains three types of nodes: NameNode (the master that keeps the namespace and metadata in memory), Secondary NameNode (a helper that periodically snapshots metadata), and DataNode (the slaves that store actual data blocks on disk).

NameNode: manages the file system namespace, handles client read/write requests, and stores metadata; it is a single point of failure.

Secondary NameNode: monitors HDFS status, saves metadata snapshots, and can serve as a standby for the NameNode.

DataNode: stores data blocks, performs block read/write operations, and provides data retrieval services.

MapReduce, proposed by Google, breaks a complex computation into a series of Map and Reduce steps, similar to the shell pipeline: cat word.txt | sort | uniq -c The MapReduce workflow can be summarized in three stages:

Map task: extracts key‑value pairs from each input line.

Sort & Shuffle: groups values by key.

Reduce task: filters, aggregates, or otherwise processes each <key, value> group.

Typical Hadoop commands (shown in the original figure) allow users to interact with the file system and launch jobs; common commands include hdfs dfs -ls, hdfs dfs -put, hadoop jar, etc.

A practical MapReduce example processes advertising click logs to produce per‑advertiser metrics. The example references three shell scripts— mapper.sh , reducer.sh , and run.sh —which implement the map, reduce, and job‑submission steps respectively.

In conclusion, readers are encouraged to study the “Hadoop: The Definitive Guide” for deeper knowledge and to explore other big‑data frameworks such as Spark, Storm, and Flink, selecting the most suitable tool based on data characteristics and performance requirements.