An Introduction to Big Data Concepts, Hadoop Ecosystem, and Common Frameworks

In the era of big data, the term has become ubiquitous, and this article introduces its basic definition: data sets that are extremely large in volume and complex in variety, making traditional databases inadequate for storage, management, and processing.

The main characteristics of big data are summarized as the 4V model—Volume, Variety, Velocity, and Veracity—with some extensions to a 6V model that adds Valence (connectivity) and Value.

The Hadoop 2.0 ecosystem is presented as a four‑layer architecture: a storage layer (HDFS), a resource and data‑management layer (YARN), a compute engine layer (MapReduce), and a query‑analysis layer (Hive, Pig). Spark is highlighted as a component that spans both the compute engine and query layers.

A direct comparison shows Hadoop as a comprehensive distributed data infrastructure providing storage, computation, and resource scheduling, while Spark is a specialized processing engine that relies on external storage systems such as HDFS.

Common big‑data concepts are grouped into six categories:

Computing frameworks: offline (Hadoop MapReduce, Spark) and real‑time (Storm, Spark Streaming, Flink).

Storage frameworks: file systems (HDFS, Tachyon, KFS) and NoSQL databases (HBase, MongoDB, Redis), plus search engines (Elasticsearch, Solr).

Resource management: YARN, Mesos.

Log collection: Flume, Logstash.

Messaging systems: Kafka, StormMQ, ZeroMQ, RabbitMQ.

Query and analysis tools: Hive, Impala, Pig, Presto, Phoenix, SparkSQL, Drill, Kylin, Druid.

Big‑data tasks are divided into offline (batch) and real‑time (streaming) jobs, each typically involving three stages: data source extraction, data transformation, and loading into the target system.

The article explains that offline tasks often use HDFS, Hive, or MySQL as sources, while real‑time tasks commonly rely on Kafka. Transformation operations include aggregations (group‑by) and joins, illustrated with a retention‑analysis example that emphasizes the importance of join and aggregation operators.

Finally, the distinction between latency (offline vs. real‑time) and processing mode (batch vs. streaming) is clarified, noting that frameworks like Spark Streaming implement micro‑batch processing to achieve near‑real‑time performance.

The article concludes by summarizing these foundational concepts as essential knowledge for anyone beginning to work with big data platforms.