Understanding Hadoop HDFS and MapReduce: Principles, Architecture, and Sample Code

The article begins with a brief history of big‑data foundations, citing Google's 2004 papers (Google File System, MapReduce, and Bigtable) that inspired Doug Cutting to create Hadoop, the open‑source framework for large‑scale data processing.

It then introduces HDFS (Hadoop Distributed File System) as the storage layer for massive data, comparing it to traditional single‑node file systems and explaining its core components: NameNode (metadata management) and DataNode (block storage). The text describes how files are split into blocks, replicated across DataNodes, and how the NameNode tracks block locations.

The read/write workflow of HDFS is detailed: a client contacts the NameNode to create a file, receives block allocation, writes blocks to DataNodes, receives acknowledgments, and finally updates metadata. Reading follows the reverse process, with the client retrieving block locations from the NameNode and streaming data from the appropriate DataNodes.

Next, the article shifts to the MapReduce computation framework, outlining its two phases—Map (transform input records into key‑value pairs) and Reduce (aggregate values for each key). A word‑count example illustrates how Map emits <word,1> pairs and Reduce sums them to produce final frequencies.

To demonstrate practical usage, the article provides complete Java source code for a Hadoop MapReduce job that processes JSON and CSV files. The Mapper parses each line, detects the format, and emits cleaned data using helper methods cleanJsonData and cleanCsvData . The Reducer simply forwards the Mapper output. The Driver class configures the job, sets the Mapper and Reducer classes, specifies input and output paths, and launches the job via Maven‑built JAR.

Code snippets are presented in their original form, wrapped in ... tags to preserve formatting. Maven dependencies required for the job include hadoop-common , hadoop-mapreduce-client-core , jackson , and commons-csv . The article concludes by encouraging readers to explore Hadoop’s source code, noting that the framework is essentially Java and that its concepts are approachable for those willing to study them.