Big Data Technology Architecture

Apr 23, 2019 · Big Data

Understanding Spark Shuffle: Stages, Evolution, and Source Code Structure

This article explains the concept of Spark Shuffle, details its two-phase write and read processes, describes the evolution from Hash‑based to Sort‑based and Tungsten‑based shuffles across Spark versions, and outlines the relevant source‑code components in Spark 2.1.