Understanding Data Skew and Its Mitigation Strategies in Distributed Computing

Data skew refers to the phenomenon where insufficient data dispersion causes a large amount of data to concentrate on one or a few machines, reducing the efficiency of distributed computation.

Analogously, data can be likened to river water and the processing nodes to hydroelectric stations; each station has a maximum capacity, and overloading one while another remains idle mirrors the impact of skew.

Data skew occurs when data is unevenly assigned to processing nodes, similar to a sudden flood that overwhelms a single dam.

1. Causes of Data Skew

Data skew is common in MapReduce frameworks and can be divided into logical and data layers:

Logical layer: aggregation, join, deduplication, etc., e.g., calculating the highest level across all characters of a player.

Data layer: inherent uneven distribution of data, such as differences between new and old game servers, weekday vs. weekend traffic, or player level distributions.

In practice, most scenarios involve both causes, making data skew a constant challenge.

2. Mitigation Methods

Preventive measures (pre‑emptive):

Split data with large volume differences and compute separately, then merge results.

Logical replacement: assign distinct placeholder values to nulls to distribute workload evenly.

Two‑stage aggregation: first aggregate per server, then aggregate across servers; useful when cross‑server aggregation is not required.

Reactive measures (after skew occurs):

Increase parallelism: raise parameters such as spark.default.parallelism or spark.sql.shuffle.partitions to spread data across more executors.

Data cleaning: filter out irrelevant, null, or abnormal records that cause skew; remove or transform low‑impact data that does not affect final results.

3. Summary

There is no universal silver bullet; the optimal solution balances current hardware, software, and business priorities. Data skew persists as data volumes grow, and new techniques continuously emerge. Ongoing learning and adaptation are essential.