What Is Data Skew in Distributed Computing, Its Impact, and How to Fix It

Interview Focus

Basic concept : Interviewers check whether you understand partition/group operations and the essence of shuffle‑induced data redistribution.

Problem diagnosis : From symptoms such as tasks stuck at 99%, OOM, or long overall execution time, infer data skew rather than network, GC, or code bugs.

Solution depth : Provide remedies for different scenarios (group by, join, count(distinct), etc.) instead of a generic “add partitions”.

Core Answer

Data skew occurs when, after a shuffle (triggered by operations like group by, join, distinct, orderBy), the majority of records for a particular key end up in a few tasks or nodes, causing severe load imbalance.

Key harm : the “bucket effect” – the whole job’s duration equals the slowest task’s duration, so a single overloaded task drags the entire job.

Solution idea : either disperse the hot keys or bypass the shuffle entirely.

Why Skew Happens

Shuffle redistributes data by hash(key) % N. If a key’s record count far exceeds others (e.g., a popular streamer’s orders occupy 30% of all orders), the partition containing that key can be dozens or hundreds of times larger than the others.

Normal distribution: 100 GB evenly split into 4 partitions (≈25 GB each) → all tasks finish together. Skewed distribution: one hot key holds 95 GB → one task processes 95 GB while the other three finish early and idle.

Because total job time ≈ slowest task time, skew amplifies latency linearly.

Problems Caused by Skew

Extended execution time : jobs often stall at the last 1 % (e.g., 99 % stuck for 40 minutes because a single user ID accounts for 60 % of data).

OOM : the overloaded reducer’s hash map or sort buffer exceeds memory limits.

Low resource utilization : only one executor runs at full load while others sit idle, wasting allocated cores.

Job failure : tasks may time out or crash due to OOM, causing the whole pipeline to restart.

Solution Framework (by Scenario)

1. Group by / Aggregation Skew – Two‑Stage Aggregation

Split a single aggregation into two phases:

Phase 1 : Add a random prefix (e.g., 0‑9) to the hot key, then perform a local aggregation (partial reduce).

Phase 2 : Remove the prefix to restore the original key and perform a global aggregation.

This is essentially an extended MapReduce combiner.

JavaPairRDD<String, Long> stage1 = data
    .mapToPair(t -> {
        int prefix = new Random().nextInt(10);
        return new Tuple2<>(prefix + "_" + t._1, t._2);
    })
    .reduceByKey(Long::sum); // local aggregation

JavaPairRDD<String, Long> stage2 = stage1
    .mapToPair(t -> {
        String originalKey = t._1.substring(t._1.indexOf("_") + 1);
        return new Tuple2<>(originalKey, t._2);
    })
    .reduceByKey(Long::sum); // global aggregation

Pitfall : count(distinct) cannot be directly split; first group by to deduplicate, then count , or use an approximate algorithm such as HyperLogLog.

2. Large Table ↔ Small Table – Map Join (Broadcast Join)

When one side is a small dimension table, broadcast it to every map task and perform the join on the map side, completely avoiding shuffle.

Broadcast<Table> smallTableBC = sc.broadcast(smallTable.collectAsMap());
JavaRDD<ResultRow> result = largeTable.map(row -> {
    Table small = smallTableBC.value();
    return joinWith(row, small.get(row.getKey())); // no shuffle
});

In Spark SQL, setting spark.sql.autoBroadcastJoinThreshold (default 10 MB) enables automatic map join.

3. Large Table ↔ Large Table – Expand + Random Prefix

When both tables are too big to broadcast:

Add a random prefix (0‑N) to the large table’s key, splitting the hot partition.

Duplicate the small table N times, each copy receiving a matching prefix.

Join on the prefixed keys; the data volume increase is limited to N ≈ 5‑10.

4. Null / Abnormal Values Concentration – Pre‑processing

If possible, filter out nulls: where user_id is not null.

Otherwise, assign a random key to nulls, e.g., concat('null_', rand()), to spread them evenly.

5. Framework‑Level Fallback – Adaptive Query Execution (AQE) / Hive Skew Switches

Spark 3.0+ AQE: enable with spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true. The engine detects large partitions at runtime and splits them into sub‑partitions.

From Spark 3.2, AQE is on by default.

Hive: set set hive.groupby.skewindata=true; (auto two‑stage aggregation) and set hive.optimize.skewjoin=true; (skew‑join optimization), optionally tuning hive.skewjoin.key (default 100 000 rows).

MapReduce: configure a combiner to reduce shuffle volume.

Typical Interview Follow‑up Questions

How to quickly identify data skew? – Check job progress stuck at 99 %, task logs for disproportionate data volume, and Spark UI/ResourceManager for a single busy executor.

Why does two‑stage aggregation work? – It first disperses the hot key with random prefixes, then aggregates locally, reducing data size before the second shuffle.

Difference between Map Join and Reduce Join? – Map Join runs on the map side with broadcasted small table (no shuffle); Reduce Join requires shuffle and matching on the reduce side.

How does Spark 3.0 AQE handle skew automatically? – It measures partition sizes at runtime, splits oversized partitions into multiple sub‑partitions, processes them in parallel, and merges results.

Memory Aid

“Diagnose, filter, broadcast if possible, disperse otherwise, fall back to framework” – choose the solution according to the scenario:

Aggregation skew → two‑stage (salt + local + global aggregation)

Small‑table join → Map Join (broadcast)

Large‑table join → expand + random prefix

Null concentration → pre‑process

Cannot change code → AQE / Hive skew switches

Summary

Data skew is fundamentally an uneven key distribution after shuffle, leading to the “bucket effect” where the slowest task stalls the whole job. The remedy is either to scatter hot keys (salt, random prefix) or to bypass shuffle (broadcast join). Mapping each scenario to its appropriate technique and leveraging Spark 3.0+ AQE provides a comprehensive, interview‑ready answer.