Data skew
Null join key holds 30% of the fact
Symptoms
- One task reads terabytes
- Business thought nulls were 0.1%
Data skew interview questions
Interactive Spark interview questions on Data skew. Same topic as /learn/spark/data-skew. When one key owns most of the data, one task becomes the stage. Salt or AQE skew join.
Question 1 of 3
beginner
Data skewWhat is data skew in Spark?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is data skew in Spark?
Data skew · tap to open the answer
Short: One key (or partition) has far more data than the others, so one task defines stage time.
Detailed: Classic: null country, 'US', or one customer_id. Median task is fine; max task is the SLA.
Common mistake: Skew as 'the cluster is unbalanced' without a key.
Follow-up: Which UI view shows skew in 10 seconds?
intermediateName three skew fixes and when each applies.
Data skew · tap to open the answer
Short: Filter/isolate hot keys, salt the key, AQE skew join.
Detailed: If nulls are junk, drop them. If one key is real (US), process it separately or salt. AQE split can help SMJ skew. Broadcast if the other side is small — skew on the fact may not matter.
Common mistake: repartition(1000) as a skew fix — the hot key still hashes to one partition.
Follow-up: Why doesn't more shuffle partitions fix a single hot key?
seniorHow do you prove skew versus a slow node?
Data skew · tap to open the answer
Short: Skew: same task id always huge shuffle read for a key. Slow node: different keys, one host.
Detailed: Look at shuffle read bytes, not just duration. If bytes are even but duration isn't, suspect disk/CPU/noisy neighbor. If bytes are skewed, it's data.
Senior: Add salt, agg/join on (key, salt), then drop salt and re-agg if needed.
Common mistake: Speculative execution as the root-cause fix.
Follow-up: Walk through salting without ruining the grain.