Interview/Spark

Spark interview questions: Partitions: the unit of parallelism

Interactive Spark interview questions on Partitions: the unit of parallelism. Practice with the matching lesson. One partition is one chunk of data. One task processes one partition in a stage. Size them for 128–256 MB.

Lesson · Simulation

What is a Spark partition?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Partitions: the unit of parallelism

Output is 12 000 tiny files

Symptoms

  • Every file is a few MB
  • Next job launches 12k tasks

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What is a Spark partition?

Partitions: the unit of parallelism · tap to open the answer

Say this first: A chunk of a DataFrame that one task reads in a stage.

Why: Input partitions often follow files. After a shuffle, spark.sql.shuffle.partitions (or AQE) decides how many. One partition = one task, not one executor.

Common mistake: One partition per executor as a rule.

Follow-up they will ask: Who decides input partitions for a Parquet scan?

Lesson · Simulation

intermediate

repartition vs coalesce — when do you use each?

Partitions: the unit of parallelism · tap to open the answer

Say this first: repartition shuffles to a new count. coalesce reduces without a full shuffle (can stay unbalanced).

Why: repartition(n) for even parallelism or a join key. coalesce(n) after a filter that dropped most rows, to avoid a shuffle. Don't coalesce to 1 on a huge frame except for a tiny output.

Common mistake: coalesce(1) to 'make a single CSV' on a 400 GB table.

Follow-up they will ask: What does partitionBy on write control versus RDD partitions?

Lesson · Simulation

senior

How do you size partitions for a shuffle?

Partitions: the unit of parallelism · tap to open the answer

Say this first: Aim for ~100–256 MB per shuffle partition, then look at the histogram.

Why: Too few: fat tasks, spill, OOM. Too many: scheduler overhead. AQE coalesce helps small data. Skew means one partition will still be fat — salting, not more partitions, fixes that.

Senior depth: You want enough partitions for cores × duration, sized by data bytes, not by machine count.

Common mistake: Setting shuffle partitions to executor_count.

Follow-up they will ask: Why is executor_count the wrong formula?

Lesson · Simulation

beginner

You point Spark at a folder of 500 small Parquet files and getNumPartitions() returns 68, not 500. Where did 68 come from?

Partitions: the unit of parallelism · tap to open the answer

Say this first: The file scan packs small files together into splits until a split reaches spark.sql.files.maxPartitionBytes.

Remember these

  • Input partitions come from byte-packed splits, not from file count
  • spark.sql.files.maxPartitionBytes caps the bytes in one input split
  • openCostInBytes charges each tiny file so they pack instead of fragment
  • One large splittable file can become many partitions

Why: Spark does not map one file to one partition. The scan planner walks the file list and bin-packs files into splits, charging each file an opening cost of spark.sql.files.openCostInBytes on top of its real size, and closes a split once it reaches spark.sql.files.maxPartitionBytes. So 500 files of a few MB each collapse into far fewer, fatter input partitions. The opposite also happens: one 3 GB splittable Parquet file becomes many partitions because a single file can be cut at row-group boundaries. If you want the real number, read it off the DataFrame with getNumPartitions or look at the task count on the FileScan stage in the Spark UI.

print(spark.conf.get("spark.sql.files.maxPartitionBytes"))
df = spark.read.parquet("/mnt/raw/events")
print(df.rdd.getNumPartitions())

Common mistake: Saying 'one file equals one partition' and then being unable to explain the 68.

Follow-up they will ask: Which of those two settings would you change to get more parallelism on the scan?

Lesson · Simulation

intermediate

Your scan stage runs 40 tasks on a cluster with 160 cores. A teammate raises spark.sql.shuffle.partitions to 800 and nothing changes. Why not?

Partitions: the unit of parallelism · tap to open the answer

Say this first: spark.sql.shuffle.partitions only sizes post-shuffle Exchanges; the scan's partition count is decided by file splitting.

Remember these

  • Input sizing and shuffle sizing are different knobs on different stages
  • Parallelism in a stage is capped by that stage's partition count
  • Lower spark.sql.files.maxPartitionBytes to split a scan finer
  • repartition after the scan works but costs a full shuffle

Why: There are two completely separate sizing paths. Input partitions are produced by the FileScan from spark.sql.files.maxPartitionBytes and the file layout, and no shuffle config touches them. Shuffle partitions are the reducer count that an Exchange targets, so spark.sql.shuffle.partitions only takes effect downstream of a groupBy, join, or repartition. With 40 input partitions and 160 cores, 120 cores sit idle for the whole scan stage no matter what the shuffle setting says. The fixes are to lower maxPartitionBytes so the files split finer, or to insert an explicit repartition right after the scan and accept the shuffle that buys you parallelism.

spark.conf.set("spark.sql.files.maxPartitionBytes", 32 * 1024 * 1024)
df = spark.read.parquet("/mnt/raw/events")
print(df.rdd.getNumPartitions())

Common mistake: Treating spark.sql.shuffle.partitions as a global parallelism dial for the whole job.

Follow-up they will ask: If the files are gzipped CSV instead of Parquet, does lowering maxPartitionBytes still help?

Lesson · Simulation

advanced

A stage runs 200 tasks on 160 cores. Is 200 a good number? Answer without guessing.

Partitions: the unit of parallelism · tap to open the answer

Say this first: Judge it by bytes per task and by how many scheduling waves it creates, not by the number itself.

Remember these

  • Target roughly 100-200 MB of shuffle read per task, then stop tuning
  • Partition count that is not a multiple of total cores leaves a ragged wave
  • Summary Metrics max vs median exposes skew that recounting cannot fix
  • Task durations in the hundreds of milliseconds mean scheduler overhead dominates

Why: Two constraints decide the count. First, bytes: open the Stage page, read Summary Metrics for Shuffle Read, and aim for the median task to land in the low hundreds of MB so tasks run for seconds rather than milliseconds or minutes. Second, waves: 200 tasks on 160 cores is one full wave plus a 40-task straggler wave, so 160 cores are busy for the first wave and 120 idle for the second. Rounding to a multiple of the core count removes that ragged tail. Then check the max-versus-median column in Summary Metrics, because an even count with a 50x max/median ratio is a skew problem that no partition count will fix.

spark.conf.set("spark.sql.shuffle.partitions", 320)  # 2 waves on 160 cores

Senior depth: With AQE on, treat the configured count as an upper bound you deliberately overshoot. Set spark.sql.shuffle.partitions high, enable spark.sql.adaptive.coalescePartitions.enabled, and let spark.sql.adaptive.advisoryPartitionSizeInBytes merge the small ones at runtime. You tune a target size once instead of a count per job.

Common mistake: Quoting '200 is the default so 200 is fine' instead of reading bytes per task off the Stage page.

Follow-up they will ask: How does AQE change this calculation?

Lesson · Simulation

senior

AQE is enabled and your post-shuffle stage still shows exactly 200 tasks. What does that tell you?

Partitions: the unit of parallelism · tap to open the answer

Say this first: Either coalescing is off, or the partitions are already at or above the advisory size so there is nothing to merge.

Remember these

  • Coalescing needs spark.sql.adaptive.coalescePartitions.enabled too
  • Look for AQEShuffleRead above the Exchange in the SQL tab to confirm it ran
  • AQE merges partitions downward; it does not create extra tasks
  • Scan stages are never coalesced by AQE

Why: AQE only reshapes partitions after a shuffle has written its map output statistics, and coalescing requires spark.sql.adaptive.coalescePartitions.enabled on top of spark.sql.adaptive.enabled. When it runs, it merges adjacent small partitions until each reaches spark.sql.adaptive.advisoryPartitionSizeInBytes, and the Spark UI SQL tab shows a CustomShuffleReader or AQEShuffleRead node above the Exchange. Seeing a clean 200 means one of three things: AQE is off for this query, the stage is not post-shuffle at all (a scan is never coalesced), or every partition is already larger than the advisory size so merging would do nothing. Note that AQE coalesces downward only — it never splits a partition to create more tasks, except in the specific skew-join path.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128MB")

Senior depth: No. Coalescing cannot split. To get smaller post-shuffle partitions you raise the configured shuffle partition count so the Exchange produces more of them, and let AQE merge back whatever turns out too small. Overshoot plus coalesce is the pattern; undershoot plus AQE is a dead end.

Common mistake: Assuming AQE is active because the cluster default says so, without checking the plan for the AQE shuffle read node.

Follow-up they will ask: Your partitions are 1 GB each and you want them smaller. Does AQE help?

Lesson · Simulation

architect

The platform team wants one cluster-wide spark.sql.shuffle.partitions for every job. What do you push back with?

Partitions: the unit of parallelism · tap to open the answer

Say this first: A fixed count is wrong for every job except the one it was tuned on, because the invariant people actually want is bytes per partition.

Remember these

  • Bytes per partition is portable across jobs; a partition count is not
  • advisoryPartitionSizeInBytes is the right platform-level default
  • Set the static count high and let AQE coalesce downward
  • Per-job overrides belong with the job code, not in cluster policy

Why: A count is a function of data volume, and volume differs by orders of magnitude across a platform. The same 200 that gives a 2 TB job sane 1 GB partitions gives a 2 GB job eight-megabyte partitions and thousands of millisecond tasks whose scheduler delay exceeds their compute. The portable default is a target size: turn on spark.sql.adaptive.enabled with coalescePartitions, set spark.sql.adaptive.advisoryPartitionSizeInBytes once as the platform standard, and set the static count generously high so AQE has material to merge. Individual jobs then override only when they have a measured reason, and that override lives in the job config next to the code, not in a cluster policy nobody can trace. The one thing a single global number does buy you is predictability for capacity planning, which is worth saying out loud before you argue against it.

Senior depth: Mine the Spark event logs. Every stage records task count, shuffle read bytes, and task durations, so you can compute median bytes per task and max/median duration ratio per stage and rank the whole platform by it. That turns partition tuning from a per-ticket argument into a dashboard with a worst-offenders list.

Common mistake: Arguing from folklore numbers instead of naming the invariant the config is standing in for.

Follow-up they will ask: How would you detect, across hundreds of jobs, which ones are badly partitioned?

Lesson · Simulation