Interview/Spark

Spark interview questions: Data skew

Interactive Spark interview questions on Data skew. Practice with the matching lesson. When one key owns most of the data, one task becomes the stage. Salt or AQE skew join.

Lesson · Simulation

beginner
Data skew

What is data skew in Spark?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Data skew

Null join key holds 30% of the fact

Symptoms

  • One task reads terabytes
  • Business thought nulls were 0.1%

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What is data skew in Spark?

Data skew · tap to open the answer

Say this first: One key (or partition) has far more data than the others, so one task defines stage time.

Why: A shuffle sends every row with the same key to the same reducer, so the partition sizes follow the key distribution and nothing rebalances them. The classic offenders are a null country, a dominant 'US', or one enterprise customer_id that accounts for a third of the orders. A stage finishes when its slowest task finishes, so 199 tasks at 4 seconds and one at 9 minutes is a 9-minute stage no matter how many executors you add. In the Stages tab the giveaway is the task duration summary: compare the max against the median, and look for Spill (Disk) on that one task.

Common mistake: Skew as 'the cluster is unbalanced' without a key.

Follow-up they will ask: Which UI view shows skew in 10 seconds?

Lesson · Simulation

intermediate

Name three skew fixes and when each applies.

Data skew · tap to open the answer

Say this first: Filter/isolate hot keys, salt the key, AQE skew join.

Why: If nulls are junk, drop them. If one key is real (US), process it separately or salt. AQE split can help SMJ skew. Broadcast if the other side is small — skew on the fact may not matter.

Common mistake: repartition(1000) as a skew fix — the hot key still hashes to one partition.

Follow-up they will ask: Why doesn't more shuffle partitions fix a single hot key?

Lesson · Simulation

senior

How do you prove skew versus a slow node?

Data skew · tap to open the answer

Say this first: Skew: same task id always huge shuffle read for a key. Slow node: different keys, one host.

Why: Look at shuffle read bytes, not just duration. If bytes are even but duration isn't, suspect disk/CPU/noisy neighbor. If bytes are skewed, it's data.

Senior depth: Add salt, agg/join on (key, salt), then drop salt and re-agg if needed.

Common mistake: Speculative execution as the root-cause fix.

Follow-up they will ask: Walk through salting without ruining the grain.

Lesson · Simulation

intermediate

AQE skew join is enabled and the skewed stage is completely unchanged. Name two reasons.

Data skew · tap to open the answer

Say this first: The partition did not clear both skew thresholds, or the operator is not a shuffle join AQE can split.

Remember these

  • Skew needs both the absolute byte threshold and the median factor
  • Only shuffle joins are split — not aggregations, not broadcast joins
  • A skew with no Exchange above it is invisible to AQE

Why: AQE calls a partition skewed only when it is both larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes and more than spark.sql.adaptive.skewJoin.skewedPartitionFactor times the median partition size. A partition that is huge in absolute terms but close to the median in a uniformly huge stage fails the second test, and a small-but-lopsided stage fails the first. The feature also applies only to joins executed as sort-merge or shuffled hash joins — it does not split a skewed groupBy, and it does not apply to a broadcast join because there is no shuffle partition to split. It needs spark.sql.adaptive.enabled on, and it works from a shuffle it can observe, so skew appearing before any Exchange is invisible to it. Check both thresholds against the actual partition sizes in the stage summary before concluding the feature is broken.

spark.conf.get("spark.sql.adaptive.skewJoin.skewedPartitionFactor")
spark.conf.get("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes")

Common mistake: Enabling skewJoin and assuming any long task anywhere in the job will now be split.

Follow-up they will ask: How would you confirm from the UI that a split actually happened?

Lesson · Simulation

advanced

Walk me through salting a join, not an aggregation. What do you do to each side?

Data skew · tap to open the answer

Say this first: Add a random salt to the skewed side's key and replicate the other side across every salt value, then join on the composite key.

Remember these

  • Skewed side gets a random salt; the other side is replicated across all salts
  • Join on (key, salt), then drop the salt from the output
  • Salting every key multiplies the other side's shuffle by N
  • Scope the salt to the known hot keys to contain the cost

Why: Salting a join is asymmetric. On the skewed side you append a random integer in a fixed range to the join key, which splits the hot key's rows across that many partitions. On the other side you cannot randomise, because a row must still meet every bucket of its key, so you explode it: cross each row with the full range of salt values so one row becomes N. Then join on key plus salt and drop the salt column from the output. The result is correct because every (key, salt) pair on the skewed side has a matching row on the exploded side. The cost is that explosion — the other side is N times larger through the shuffle — so scope the salt to the known hot keys rather than applying it to every key, typically with a conditional salt that is zero for everything else.

N = 16
f = fact.withColumn("salt", (F.rand() * N).cast("int"))
d = dim.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
f.join(d, ["key", "salt"]).drop("salt")

Senior depth: From the ratio you need to fix, not from a round number. Take the hot partition's Shuffle Read and divide by the median partition size — that quotient is roughly the N that flattens it. A larger N splits the hot key further but multiplies the replicated side, so there is a point where the explosion costs more than the straggler. Measure max against median Shuffle Read after the change rather than assuming it worked.

Common mistake: Salting only the fact side and losing every row whose key hashed to a salt the dimension does not have.

Follow-up they will ask: How do you choose N?

Lesson · Simulation

advanced

The same hot key makes a join unusable but a groupBy().count() on it finishes fine. Why the difference?

Data skew · tap to open the answer

Say this first: Aggregation does a partial pass on the map side, so the hot key reaches the reducer already collapsed; a join has to move every row.

Remember these

  • The partial HashAggregate collapses the hot key before the shuffle
  • A join must move every row of the hot key across the Exchange
  • collect_list, collect_set, and distinct defeat the partial pass

Why: A reducing aggregation plans as two HashAggregate nodes around the Exchange: the partial one combines rows per key within each input partition, so a hot key contributes one partial row per partition instead of millions of raw rows. By the time the shuffle happens there is very little data for that key, and the reduce-side HashAggregate finishes quickly. A join cannot do that, because matching requires the actual rows, so every row of the hot key crosses the shuffle into one partition and one task hashes or sorts all of it. The exception is an aggregation that cannot combine incrementally: collect_list, collect_set, and count(DISTINCT ...) without a rewrite keep per-key data proportional to the key's row count, so they skew exactly like a join. Read the plan — two HashAggregate nodes means the map-side combine is happening, one means it is not.

df.groupBy("key").count().explain("formatted")
# two HashAggregate nodes around the Exchange = map-side combine

Senior depth: Stop collecting the full list. Either aggregate to the statistic you actually need so the partial pass can combine it, cap the list with a windowed top-N before collecting, or branch the hot key into its own job. If the full list is genuinely required, two-stage it: collect per (key, salt), then merge the lists per key, which keeps any single task's memory bounded.

Common mistake: Treating skew as one phenomenon and expecting identical symptoms from aggregations and joins.

Follow-up they will ask: How would you make a skewed collect_list survivable?

Lesson · Simulation

senior

Every shuffle stage is even, but the final write has one task running for fifty minutes. What is going on?

Data skew · tap to open the answer

Say this first: A partitioned write funnels all rows for one output partition value into one task, so a skewed partition column becomes a skewed writer.

Remember these

  • partitionBy funnels one output value into one writing task
  • Signature: even shuffle stages, one long task in the write stage
  • maxRecordsPerFile bounds the output a single task writes per file
  • A skewed partition column is a layout mistake, not a tuning problem

Why: With partitionBy on the write, Spark wants each output directory written by as few tasks as possible, so plans typically include a shuffle or a sort on the partition columns; one hot value then lands in one task that writes the entire directory. Dynamic partition overwrite and sorted writes amplify it, because the task may also hold or sort rows per open file. The signature is clean shuffle stages followed by a write stage whose task duration histogram has a single long bar, high Spill (Disk) on that task, and one output file far larger than the rest. The cheapest fix is to repartition on the partition columns plus a bounded salt before the write so the hot value is split across a known number of files, or to set maxRecordsPerFile so one task cuts its output into bounded files. If the partition column is badly skewed or high cardinality, the real answer is to stop partitioning by it.

N = 8
(df.withColumn("salt", (F.rand() * N).cast("int"))
   .repartition("event_date", "salt").drop("salt")
   .write.partitionBy("event_date").save(path))

Senior depth: Splitting a hot value across N writers produces N files for that partition instead of one, so you trade a straggler for small-file pressure. Size N from the target file size — divide the partition's bytes by roughly 128 MB to 1 GB depending on the reader — and compact on a schedule. Liquid clustering and OPTIMIZE exist precisely so you do not have to choose between a straggler and a thousand tiny files forever.

Common mistake: Looking only at shuffle stages for skew and ignoring the write stage's task histogram.

Follow-up they will ask: What does this do to the number of output files?

Lesson · Simulation

architect

The hot key is 40% of volume, the business will not change the key, and salting is already in place. Design the pipeline.

Data skew · tap to open the answer

Say this first: Split the pipeline by key class: a dedicated path for the hot keys and the ordinary path for the long tail.

Remember these

  • At 40% of volume it is two workloads, not one unbalanced shuffle
  • The hot path gets its own parallelism, layout, and incremental strategy
  • Classify hot keys from profiling data, not from a hard-coded list
  • Add a reconciliation step so the two paths provably sum to the whole

Why: Once a single key is a large share of volume you are not balancing a shuffle any more, you are running two workloads in one job. Isolate the hot keys into their own branch with their own parallelism, their own output layout, and where the computation allows a pre-aggregation or incremental model, so the full history is never reshuffled. Run the long tail through the ordinary path where the hash partitioner behaves. The hot path can also change shape: a key that is 40% of a fact usually supports incremental maintenance — process only new rows and merge into a stored aggregate — instead of a daily full recompute. Encode the key classification as data rather than a hard-coded list, so a new hot key is detected by a profiling job and routed automatically. The cost is two code paths to maintain and a reconciliation step to prove they add up.

Senior depth: Profile the join and groupBy keys as a scheduled job: rows per key, and the max-to-median ratio of the resulting partition sizes. Alert when a key crosses the share your current salt factor can absorb. That turns skew from an on-call page into a backlog item, and it also gives the hot-path routing table its input.

Common mistake: Increasing the salt factor every quarter instead of admitting the pipeline needs two paths.

Follow-up they will ask: How do you detect the next hot key before it causes an incident?

Lesson · Simulation