Day 39: Big Data Architecture – Distributed Storage, Batch vs Stream Processing, and Compute‑Storage Separation

Why a Single Database Fails

When data grows from gigabytes to terabytes or petabytes, a single relational database hits limits: storage capacity, write throughput, unsuitable data types (e.g., images, logs), query contention, CPU/memory/disk ceilings, and single‑point‑of‑failure risk. Adding cache, queues, or read/write splitting only postpones the need to manage sharding, migration, replication, fault recovery, and cross‑shard computation, shifting complexity from the DB to the application and operations.

Distributed Storage Fundamentals

Sharding – split large files or datasets into blocks and place each block on a different node (e.g., blocks A‑D on nodes 1‑4).

Replication – keep multiple copies of each block on separate nodes to survive failures.

Horizontal scaling (Scale‑Out) – add more commodity nodes (3 → 6 → 20) to increase capacity and throughput; contrast with vertical scaling (Scale‑Up) which upgrades a single machine.

In Hadoop Distributed File System (HDFS) the two roles are:

NameNode: manages metadata, knows which blocks belong to which files and where they reside.</code><code>DataNode: stores the actual data blocks and serves read/write requests.

Data locality means scheduling compute tasks on the nodes that hold the required blocks, minimizing network transfer.

Batch Processing

Batch processing works on bounded datasets (e.g., yesterday’s orders, a year’s sales ranking, five‑year log analysis). Typical flow:

Collect → Clean & Join → Aggregate by dimensions → Generate report

Key characteristics: full‑volume, high accuracy, high throughput, and tolerance for minute‑ to hour‑level latency. Common engines are MapReduce, Spark (batch mode) and Hive, which provides an SQL‑like interface that the engine translates into distributed jobs.

Example Hive query for yesterday’s sales:

SELECT province, SUM(amount) FROM orders WHERE order_date = '2026-07-27' GROUP BY province;

Hive exposes a table‑like schema on top of files stored in HDFS, then generates MapReduce/Tez/Spark jobs to execute the query.

Stream Processing

Stream processing handles unbounded, continuously arriving data (clicks, real‑time orders, sensor events). Processing steps:

Event arrives → Filter/Transform/Aggregate → Update result within seconds or milliseconds

Key concerns: low latency, incremental computation, ordering, and fault‑tolerant replay.

Storm topology example:

Kafka Spout → Filter Bolt → Aggregate Bolt → Alert Bolt

Storm processes each event as it arrives, keeping the topology running indefinitely. Flink offers native stream processing with state, windows, and event‑time handling. Spark Streaming (micro‑batch) groups events into small batches (e.g., 1‑second windows) and processes them with Spark’s batch engine.

Batch vs. Stream Comparison

Data : Batch – bounded historical set; Stream – continuous event stream.

Latency : Batch – minutes to hours; Stream – seconds to milliseconds.

Computation : Batch – full‑volume, re‑run possible; Stream – incremental, always‑on.

Focus : Batch – accuracy, throughput, re‑calc; Stream – real‑time, ordering, continuity.

Typical use‑cases : Batch – next‑day reports, historical analytics; Stream – live dashboards, monitoring, alerts.

Tech examples : Batch – MapReduce, Spark, Hive; Stream – Flink, Storm, Spark Streaming.

Component Responsibilities

Kafka : ingest, buffer and deliver events; supports replay.

HDFS : durable, distributed storage of raw files.

Spark : large‑scale batch computation, interactive analysis, ML.

Flink : low‑latency, stateful stream processing.

Compute‑Storage Integration vs. Separation

Integration (e.g., classic Hadoop)

Pros: reduces network traffic, high throughput when data and compute co‑locate.

Cons: compute and storage must scale together; limited elasticity; upgrades tightly coupled.

Separation

Pros: independent scaling of storage and compute, multiple engines share the same data, elastic compute for spikes.

Cons: relies on high‑speed network, incurs data transfer cost, adds complexity for caching, permissions and format compatibility.

Choosing between them depends on data size, network cost, workload variability, and elasticity requirements.

Fault Tolerance in Distributed Systems

When a node fails, the system follows this chain:

Node failure detected → Read from remaining replicas → Reschedule failed tasks to healthy nodes → Replay input or recompute if needed → Re‑balance replicas → Verify restored state.

This mirrors high‑availability principles: redundancy alone is insufficient without detection, failover, and validation.

End‑to‑End E‑Commerce Data Platform Example

Browse/Click/Order/Payment → Kafka (buffer) → HDFS / Object Store (raw) →
  Batch: Spark / MapReduce → Daily report
  Stream: Flink (or Storm) → 5‑second dashboard & 10‑second alerts
Result storage (DB / cache) → Dashboards, reports, business services

Key design points:

Store raw events with sharding, replication, and horizontal scaling.

Use stream processing for real‑time metrics and alerts.

Use batch processing for accurate next‑day reports.

Adopt compute‑storage separation to allow elastic compute during promotions while keeping data permanently in distributed storage.

Ensure fault tolerance via replicas, task retry, and data replay.