Understanding Hadoop: Architecture, HDFS, MapReduce, and Their Pros & Cons

Hadoop, developed by the Apache Foundation, became the de facto platform that lowered the barrier for companies to adopt big‑data technologies, which previously were limited to giants like Google and Amazon.

Distributed File System (HDFS)

HDFS is the core storage component of Hadoop. It splits large files into 128 MB blocks, distributes them across multiple DataNodes, and maintains a NameNode that tracks block locations. By default each block is replicated on three nodes, providing parallel read performance and fault tolerance without the need for separate backups.

Distributed Computing Framework (MapReduce)

MapReduce is Hadoop’s processing engine. Input data is first split across nodes, then each node performs a Map step that emits key‑value pairs (e.g., word → 1 for word‑count). The Shuffle phase groups identical keys across nodes, which is the most network‑intensive part. Finally, each node runs a Reduce step that aggregates values (summing the ones) to produce the final result.

Advantages and Disadvantages

MapReduce offers massive parallelism and high throughput for batch processing of large datasets.

It lacks indexing; every job scans the entire input, making it inefficient for queries that touch only a small fraction of data.

Best suited for offline, bulk analytics rather than interactive, low‑latency queries or transactional workloads.

Works well with large files; processing many small files requires combining them into larger containers (e.g., SequenceFiles) to achieve good performance.

Overall, Hadoop’s ecosystem—including Hive, Storm, Mahout, HBase, ZooKeeper, Sqoop, and Flume—builds on HDFS and MapReduce to provide storage, batch processing, streaming, machine‑learning, and data ingestion capabilities for a wide range of big‑data applications.