Core Big Data Technologies Explained with a Restaurant Analogy

Big data technologies span six major stages: data collection, storage, computation, analysis, mining, and visualization. The author uses a restaurant analogy, where data are ingredients that must be gathered, stored, processed, and served as dishes.

1. Data Ingestion

Tools that bring scattered data into the system, like a conveyor belt for ingredients:

Flume / Logstash – collect server logs.

Kafka – buffer fast‑incoming data (message queue).

Sqoop – transfer data between traditional databases (e.g., MySQL) and Hadoop.

Web crawlers – scrape data from websites.

2. Distributed Storage

After gathering, data need a place to reside that can scale across many machines:

HDFS – Hadoop’s distributed file system, stores large files in chunks with triple replication.

HBase – fast random read/write for massive data (e.g., user profiles), like a shelf you can pull items from anytime.

Object storage (S3, OSS) – stores unstructured data such as images and videos.

3. Distributed Resource Management

Coordinating thousands of machines (chefs, stoves) to allocate CPU and memory efficiently:

YARN – Hadoop’s resource manager.

Mesos – another resource‑management framework.

Kubernetes (K8s) – cloud‑native container orchestration platform.

4. Data Computation

Processing raw ingredients into semi‑finished dishes, divided into two categories:

Batch processing – MapReduce (stable but slower) and Spark (in‑memory, dozens of times faster than MapReduce).

Stream processing – Flink (real‑time), Storm / Spark Streaming (also handle streams).

5. Data Analysis

Extracting valuable patterns from processed data to support decisions (e.g., which cuisine sells best on Friday):

Data warehouses & OLAP – Hive, ClickHouse for multi‑dimensional analytics.

Data mining / Machine learning – Spark MLlib, TensorFlow for tasks like sales forecasting, user segmentation, recommendation.

6. Task Scheduling

Automating the workflow order (wash → cut → cook) with dependencies and timing:

Azkaban / Oozie – Hadoop ecosystem schedulers.

DolphinScheduler / Airflow – modern workflow platforms supporting complex dependencies, retries, and timed triggers.

7. Underlying Infrastructure

Foundational services that keep the whole system running, analogous to utilities and foundations:

Distributed coordination – Zookeeper manages node status, leader election, and consistency.

Serialization – Avro, Protobuf for efficient network data transfer.

RPC communication – enables machines to talk efficiently.

Consensus protocols – Paxos, Raft ensure data synchronization across nodes.

8. Data Retrieval

Fast search over massive datasets, similar to quickly finding a menu item:

Elasticsearch – full‑text search engine using inverted indexes for millisecond queries.

Solr – another full‑text search engine.

9. Cluster Installation & Operations

Managing dozens to hundreds of machines, installing, monitoring, upgrading, and troubleshooting:

Cloudera Manager / Ambari – one‑click deployment and visual management of big‑data clusters.

Prometheus + Grafana – real‑time monitoring of CPU, memory, disk, with alerting.

Ansible – automation of configuration and batch management.

Overall, the core ideas are divide‑and‑conquer, redundancy for fault tolerance, and unified resource scheduling to maximize utilization across the big‑data ecosystem.