Tagged articles
Shuffle
65 articles · Page 1 of 1
This article provides a comprehensive technical deep dive into Ray Data, covering its architecture, streaming execution model, shuffle implementation, fault tolerance, performance tuning, and best practices for AI data pipelines, positioning it as a superior alternative to Spark for GPU-intensive workloads.
AI Data PipelinesPerformance TuningRay
0 likes · 26 min read

Learn how to use PHP's built-in shuffle() function to randomly reorder array elements, with clear syntax explanation, step-by-step code examples, output demonstration, and important considerations such as in‑place modification, handling associative or multidimensional arrays, and preserving original data when needed.
BackendShufflearray
0 likes · 3 min read

Learn how to randomize array order in PHP using built-in functions like shuffle() and array_rand(), with clear code examples and a practical scenario of assigning students to classes, plus step-by-step explanations of each method’s behavior and usage.
PHPShufflearray
0 likes · 2 min read

This article details practical Spark performance optimizations—including RDD reuse, broadcast variables, Kryo serialization, parallelism settings, shuffle tuning, and JVM tweaks—while also presenting systematic solutions for data skew, shuffle failures, serialization errors, and YARN mode issues, all illustrated with concrete code snippets and examples.
BroadcastData SkewJVM
0 likes · 25 min read

This article explains the PHP shuffle() function, detailing its syntax, behavior of modifying the original indexed array, return value, usage with both indexed and associative arrays, and provides multiple code examples demonstrating random reordering and the effect on array keys.
BackendPHPShuffle
0 likes · 5 min read

This article explains PHP's shuffle() function, detailing its syntax, behavior of modifying the original array, return value, usage with indexed and associative arrays, and provides multiple code examples illustrating how to randomize array elements.
PHPShuffleTutorial
0 likes · 4 min read

LinkedIn’s massive Spark workloads suffer from shuffle bottlenecks caused by tiny shuffle blocks, unreliable RPC connections, and data skew, so the authors design Magnet—a push‑merge shuffle service that merges blocks into large chunks, improves disk I/O, tolerates failures, and cuts end‑to‑end job time by nearly 30% regardless of hardware.
Disk I/O optimizationPush‑based serviceShuffle
0 likes · 56 min read

This article explains the PHP shuffle() function, detailing its syntax, behavior of modifying the original indexed array, return value, and provides multiple code examples—including handling of non-indexed arrays—to demonstrate how to randomly reorder array elements in PHP.
BackendFunctionPHP
0 likes · 4 min read

This article explains PHP's shuffle() function, its syntax, behavior on indexed and associative arrays, and provides code examples demonstrating how to randomize array elements and handle the function's boolean return value in practice.
BackendShufflearray
0 likes · 5 min read

This article examines the critical role of shuffle in big‑data processing, compares Spark's native shuffle with the External Shuffle Service (ESS) and Remote Shuffle Service (RSS) solutions, introduces Uniffle's architecture and configuration, and shares practical deployment experiences and performance results within a 360 internal environment.
External Shuffle ServiceRemote Shuffle ServiceShuffle
0 likes · 15 min read

This article details Zhihu's migration of massive Spark and MapReduce shuffle workloads from the External Shuffle Service (ESS) to a push‑based Remote Shuffle Service (RSS) powered by Celeborn, covering background problems, evaluation of open‑source implementations, deployment architecture, encountered issues, solutions, performance gains, and future plans.
RSSShuffleSpark
0 likes · 19 min read

This article details ByteDance's large‑scale evolution of Spark Shuffle to a cloud‑native architecture, describing background, stability and mixed‑resource scenarios, challenges such as CPU and I/O limits, custom ESS enhancements, shuffle throttling, spill‑split mechanisms, and the Cloud Shuffle Service with its push‑based design and performance gains.
KubernetesShuffleSpark
0 likes · 21 min read

Facing exploding data volumes and rising cluster costs, HuoLala adopted Apache Tez’s Remote Shuffle Service built on Apache Uniffle, redesigning the Tez client to operate without source modifications, detailing architecture, implementation challenges, testing, stability measures, and future plans to enhance big‑data shuffle performance and cost efficiency.
Apache TezData EngineeringRemote Shuffle Service
0 likes · 14 min read

This article explains the PHP shuffle function, its syntax, how it directly modifies an array to randomize element order, provides example code with output, and discusses important considerations such as preserving the original array and handling associative or multidimensional arrays.
PHPShufflearray
0 likes · 3 min read

This article explains the PHP shuffle() function, demonstrates how to create arrays, use shuffle() to randomize their elements, display the results, and shows additional uses such as generating random numbers with range() and shuffle(), providing clear code examples throughout.
BackendPHPShuffle
0 likes · 4 min read

This article examines ODPS SQL performance by dissecting logical execution plans and Logview visualizations, explaining the underlying principles of various optimization techniques such as multi‑distinct handling, shuffle reduction, system parameters, and different join strategies, and demonstrates how to apply these methods to improve query efficiency in real‑world data engineering tasks.
ODPSShuffleexecution plan
0 likes · 17 min read

This article explains how to use PHP's built-in shuffle() function to randomly reorder array elements, covering its syntax, return value, example code for indexed and associative arrays, handling of multidimensional arrays, and important considerations such as in‑place modification and preserving original data.
BackendPHPShuffle
0 likes · 3 min read

Facing massive daily Spark shuffle volumes and unstable ETL performance, ZTO Express migrated from the community External Shuffle Service to Celeborn's Remote Shuffle Service, achieving higher disk I/O efficiency, better reliability, reduced network connections, and significant reductions in task failures and job latency.
Remote Shuffle ServiceShuffleSpark
0 likes · 15 min read

This article explains PHP's shuffle() function, its syntax, behavior, return value, and demonstrates how it randomizes both indexed and associative arrays with code examples, highlighting that it modifies the original array and reindexes non‑sequential keys.
BackendShuffleTutorial
0 likes · 5 min read

This article details ByteDance's extensive migration of Spark Shuffle to a cloud‑native architecture, describing the massive data volumes, the underlying ESS and CSS services, the challenges of resource isolation, monitoring, throttling, spill‑splitting, and the performance gains achieved across stable and mixed‑resource clusters.
ByteDanceShuffleSpark
0 likes · 20 min read

This article explains the PHP shuffle() function, detailing its syntax, return behavior, usage examples, and important considerations such as its effect on the original array, limitations with associative arrays, and handling of duplicate elements, providing a practical code demonstration.
PHPShufflearray
0 likes · 3 min read

This article explains the concept of data skew in offline big‑data jobs, describes its symptoms and root causes, and provides practical optimization techniques for Hive and Spark—including partitioning strategies, map‑join usage, adaptive query settings, and monitoring approaches—to prevent performance degradation and runtime failures.
Data SkewHiveSQL
0 likes · 17 min read

This article explains the causes of Spark data skew, how to locate skewed tasks using the Web UI, and presents six optimization methods—including increasing shuffle parallelism, filtering abnormal keys, two‑stage aggregation, map‑join, key sampling, and random‑prefix joins—plus a real‑world case study.
Data SkewShufflebig data
0 likes · 21 min read

This article details Bilibili's evolution of its offline computing platform from Hadoop‑based Hive to Spark, describing the migration process, automated SQL conversion, result verification, stability and performance enhancements, meta‑store optimizations, and future work on remote shuffle and vectorized execution.
Data SkippingHiveMetaStore
0 likes · 28 min read

This article explains the evolution and inner workings of Spark's shuffle phase, comparing the original Hash‑based shuffle, the default Sort‑based shuffle, the optimized Tungsten‑Sort shuffle, and related configuration options that affect performance and file handling in large‑scale data processing.
Hash ShuffleShuffleSort-Shuffle
0 likes · 17 min read

This article provides an in‑depth guide to Spark performance optimization, covering the ten development principles, static and unified memory models, resource parameter tuning, data skew detection and mitigation techniques, as well as shuffle‑related configuration adjustments, supplemented with practical code examples and diagrams.
Data SkewPerformance TuningShuffle
0 likes · 40 min read

This article presents a comprehensive list of essential Spark configuration settings—including executor memory, off‑heap memory, memory fractions, shuffle options, and adaptive query execution parameters—each accompanied by a concise description to help users fine‑tune Spark performance.
ShuffleSparkadaptive query execution
0 likes · 6 min read

This article provides a comprehensive overview of Spark's shuffle process, explaining its definition, internal mechanisms such as shuffle write and read, the evolution of shuffle managers, and practical optimization techniques including parameter tuning and broadcast variables, all aimed at improving performance in large‑scale data processing.
ShuffleShuffle ReaderShuffle Writer
0 likes · 18 min read

This article explains HDFS reliability features such as replica policies, rack awareness, heartbeat, safe mode, checksums, trash, metadata protection and snapshots, then details YARN failover handling for ApplicationMaster, NodeManager and ResourceManager, and finally describes the Hadoop MapReduce shuffle workflow and tuning tips.
HDFSMapReduceShuffle
0 likes · 13 min read

This article explains how Hadoop's MapReduce shuffle uses a circular buffer data structure to store serialized key/value pairs and their metadata in memory, describes its initialization, write path, spill handling, and the underlying algorithms that ensure efficient in‑memory sorting and disk spilling.
HadoopIn-Memory BufferMapReduce
0 likes · 24 min read

This article presents a detailed collection of Spark performance‑tuning techniques—including submit‑script parameters, RDD and operator optimizations, parallelism and memory settings, broadcast variables, Kryo serialization, locality wait adjustments—as well as systematic methods for detecting and resolving data skew and common runtime issues such as shuffle failures, serialization errors, and JVM memory problems.
Data SkewJVM tuningSerialization
0 likes · 21 min read

This article provides a detailed collection of Spark interview questions covering deployment modes, performance advantages over MapReduce, shuffle mechanisms, RDD characteristics, optimization techniques, resource management, and various practical aspects of Spark on YARN, Mesos, and Kubernetes.
InterviewRDDShuffle
0 likes · 21 min read

This article explains how Spark coordinates master, worker, driver, and executor components to generate, submit, and run jobs, detailing the creation of logical and physical execution graphs, task allocation, result handling, and the shuffle read process with code examples and diagrams.
Job ExecutionShuffleSpark
0 likes · 14 min read

This guide compiles practical Spark tuning techniques, covering essential configuration parameters, programming best‑practices, detailed shuffle mechanics, and join optimization strategies, while also addressing common errors and mitigation steps, enabling developers to improve performance and resource utilization in large‑scale data processing jobs.
ShuffleSparkbig data
0 likes · 25 min read

This article explains how Spark implements shuffle write and shuffle read, compares its high‑level and low‑level processes with Hadoop MapReduce, and details the internal data structures, memory‑disk trade‑offs, and configuration options that affect performance.
MapReduceMemoryManagementRDD
0 likes · 21 min read

This article explains how Spark builds the logical execution graph for RDDs, describes the four-step job processing pipeline, details the various dependency types such as NarrowDependency and ShuffleDependency, and reviews common transformations and their data‑flow characteristics.
RDDShuffleSpark
0 likes · 19 min read

This article explains advanced Spark performance tuning techniques, focusing on diagnosing and resolving data skew and shuffle bottlenecks through stage analysis, key distribution inspection, and a variety of practical solutions such as Hive pre‑processing, key filtering, parallelism increase, two‑stage aggregation, map‑join, and combined strategies, while also covering ShuffleManager internals and related configuration parameters.
Data SkewPerformance TuningScala
0 likes · 47 min read

This article provides a comprehensive guide to Spark performance optimization, covering development‑level tuning principles, resource configuration parameters, practical code examples, and best‑practice recommendations to achieve high‑throughput big‑data processing.
RDDResource TuningShuffle
0 likes · 33 min read

This guide presents a complete Spark performance optimization handbook covering development‑time best practices, resource‑parameter tuning, detailed data‑skew detection and mitigation techniques, advanced shuffle‑engine configurations, and practical code examples to help engineers build faster, more reliable Spark jobs.
Data SkewResource TuningShuffle
0 likes · 69 min read

This article explains the PHP shuffle() function, describing its purpose of randomly reordering array elements, the required array parameter, the boolean return value, and provides a complete example with sample output to illustrate its usage.
BackendPHPShuffle
0 likes · 2 min read

This article provides a comprehensive guide to Spark performance tuning, covering essential configuration parameters, best‑practice programming recommendations, detailed shuffle mechanics, join optimization strategies, and common error troubleshooting for big‑data workloads.
ParametersPerformance TuningShuffle
0 likes · 20 min read

This article showcases seven ultra‑compact yet powerful code examples—from a zero‑code deployment tool and a two‑line shuffle algorithm to sleep sort, a one‑line Python AI snippet, a simple tomorrow‑time sleep call, the legendary fast inverse square‑root constant, and the classic hello‑world program.
AlgorithmsFast Inverse Square RootShuffle
0 likes · 6 min read


ITPUBNov 16, 2020 · Fundamentals
This article showcases seven intriguing code tricks—from a zero‑code deployment project and a concise shuffle algorithm to a sleep‑sort implementation, a one‑line AI chatbot, a simple next‑day timer, the legendary fast inverse square‑root constant, and the classic hello‑world example—each illustrated with brief explanations and runnable snippets.
AlgorithmsFast Inverse Square RootJava
0 likes · 6 min read

This article explains the purpose, default values, and practical tuning recommendations for common Spark submit options such as executor counts, memory settings, shuffle behavior, speculation, and various Spark SQL configurations to help users optimize big‑data workloads.
ConfigurationExecutorPerformance Tuning
0 likes · 14 min read

This article explains the PHP shuffle() function, its syntax, parameters, return values, and demonstrates three practical examples showing how to randomize indexed and associative arrays, highlighting the effect on array keys and values after shuffling.
ShuffleTutorialarray
0 likes · 3 min read

This article explains the concept and workflow of shuffle in Hadoop MapReduce and Spark, covering map‑side buffering, spill and merge, reduce‑side copy‑merge‑reduce, the reasons for sorting and file merging, and compares Hash‑Shuffle and Sort‑Shuffle implementations with performance considerations.
Hash ShuffleShuffleSort-Shuffle
0 likes · 16 min read

This article presents a detailed summary of Meituan's Spark optimization techniques, covering development‑level RDD tuning, resource parameter configuration, data‑skew mitigation, shuffle improvements, and the advantages of using DataFrame/Dataset APIs for better performance.
Performance TuningShuffleSpark
0 likes · 12 min read

This article explains what data skew is in distributed big‑data systems like Spark and Hadoop, why it hurts performance, how to spot it using the Web UI or key statistics, and presents eight practical mitigation techniques ranging from filtering and shuffle parallelism to custom partitioners and broadcast joins.
Broadcast JoinData SkewHadoop
0 likes · 19 min read

This article explains how Hadoop's MapReduce shuffle uses a circular buffer to store serialized key/value pairs and their metadata, detailing its structure, initialization, write path, spill logic, and the background thread that sorts and writes data to disk.
HadoopJavaMapReduce
0 likes · 24 min read

This article provides an in‑depth, step‑by‑step guide to optimizing Spark jobs, covering development‑time best practices, resource‑parameter tuning, data‑skew detection and mitigation techniques, and shuffle‑stage performance tweaks, complete with Scala code examples and practical recommendations.
Data SkewResource TuningScala
0 likes · 67 min read

Data skew in Spark and Hadoop occurs when a few keys dominate shuffle traffic, causing slow tasks, OOM errors, and job failures; the article describes how to detect skew via UI metrics or sampling and offers mitigation tactics such as filtering keys, increasing shuffle partitions, custom partitioners, broadcast joins, salted keys, and Hadoop‑specific settings.
Data SkewPartitioningShuffle
0 likes · 18 min read

This article explains the evolution of Spark Shuffle from hash‑based to sort‑based, introduces the Smart Shuffle optimization, details their implementations and configurations, and presents performance comparisons using TPC‑DS benchmarks, highlighting significant speedups and reduced I/O overhead.
ShuffleSmart ShuffleSort-based Shuffle
0 likes · 7 min read

This article analyzes three Spark shuffle bottlenecks—oversized partitions that exceed Netty's 2 GB limit, excessive retry latency caused by dead executors, and insufficient data‑corruption checks—and presents concrete configuration changes, new block identifiers, executor‑liveness checks, and CRC‑32 verification to improve fetchability, efficiency, and reliability at scale.
ShuffleSparkbig data
0 likes · 18 min read

This article explains the phenomenon, causes, detection methods, and a comprehensive set of solutions—including Hive preprocessing, key filtering, shuffle parallelism, two‑stage aggregation, map‑join, sampling, random prefixing, and combined strategies—to mitigate data skew in Spark jobs and improve performance.
Data SkewShuffleSpark
0 likes · 31 min read

The article explains how Flink's shuffle pipeline—from upstream data serialization to downstream consumption—is optimized through a credit‑based flow‑control mechanism, zero‑copy network buffers, broadcast serialization reduction, external shuffle service, and a plugin‑based shuffle manager, resulting in significant performance gains for both streaming and batch jobs.
FlinkFlow ControlSerialization
0 likes · 15 min read

The article explains Hadoop's MapReduce framework as both a programming model and execution engine, detailing its map and reduce phases, the WordCount example code, job startup components, data shuffling, partitioning, and how large‑scale distributed computations are orchestrated across a cluster.
HadoopMapReduceShuffle
0 likes · 10 min read

This article explains how to improve Spark join performance by reducing shuffle, using appropriate partitioners, applying broadcast hash joins for small tables, and selecting the optimal join strategy (broadcast, shuffle hash, or sort‑merge) in both Spark Core and Spark SQL.
BroadcastShuffleSpark
0 likes · 6 min read

This article examines Spark's memory management and the shuffle process, identifies the components that consume the most memory during shuffle write and read, analyzes common OOM scenarios such as task concurrency and data skew, and offers configuration tips to prevent out‑of‑memory failures.
MemoryManagementOutOfMemoryShuffle
0 likes · 14 min read

This article examines the historical evolution of Spark's shuffle implementations—from early Hash‑Based Shuffle to modern SortShuffleWriter, BypassMergeSortShuffleWriter, and UnsafeShuffleWriter—explaining their design choices, selection criteria, and the corresponding shuffle reader architecture in a production‑grade Spark 2.1.1 environment.
ShuffleShuffle WriterSpark
0 likes · 13 min read


21CTOMay 17, 2018 · Big Data
This article explains Hadoop's core components, the MapReduce programming model, the detailed shuffle and merge processes, and how YARN replaces the classic JobTracker/TaskTracker architecture to improve scalability and resource utilization in large‑scale data processing clusters.
HadoopShuffleYARN
0 likes · 12 min read


ITPUBMar 29, 2018 · Big Data
This article explains Hadoop’s core components, the MapReduce programming model, the detailed shuffle and merge processes, and how YARN replaces the classic JobTracker/TaskTracker design to improve scalability and resource utilization in large‑scale data processing clusters.
HadoopMapReduceShuffle
0 likes · 15 min read

This article explains what Spark data skew is, why it hurts performance, and presents six practical mitigation techniques—including adjusting parallelism, custom partitioners, map‑side joins, and adding random prefixes—backed by detailed experiments, code snippets, and performance comparisons.
Data SkewMap-side JoinPartitioner
0 likes · 18 min read

The article explains MapReduce’s divide‑and‑conquer model and Hadoop’s execution pipeline—including map, partition, spill, merge, shuffle, and reduce phases—illustrated with a WordCount example that shows how mappers emit word‑1 pairs and reducers aggregate counts to produce final frequencies on HDFS.
HadoopMapReduceShuffle
0 likes · 7 min read

The article details the author’s experience upgrading a production Spark cluster from version 1.4.1 to 1.6.1, exposing memory‑spill, unified memory, BlockManager deadlock, Yarn‑kill, UI quirks, and Spark‑SQL compatibility issues, and proposes concrete code‑level fixes for each problem.
ShuffleSparkYARN
0 likes · 14 min read

This article reviews Spark 1.2’s major enhancements—including the External Data Source API, column pruning, predicate pushdown, and in‑memory columnar storage—while also detailing Baidu’s large‑scale Spark deployments, its custom high‑performance Shuffle service, and the integration of Spark with the Tachyon memory file system.
BaiduExternal Data Source APIIn-Memory Columnar Storage
0 likes · 16 min read
