Partitions: the unit of parallelism interview questions

Short: A chunk of a DataFrame that one task reads in a stage.

Detailed: Input partitions often follow files. After a shuffle, spark.sql.shuffle.partitions (or AQE) decides how many. One partition = one task, not one executor.

Common mistake: One partition per executor as a rule.

Follow-up: Who decides input partitions for a Parquet scan?

Lesson · Simulation

Short: repartition shuffles to a new count. coalesce reduces without a full shuffle (can stay unbalanced).

Detailed: repartition(n) for even parallelism or a join key. coalesce(n) after a filter that dropped most rows, to avoid a shuffle. Don't coalesce to 1 on a huge frame except for a tiny output.

Common mistake: coalesce(1) to 'make a single CSV' on a 400 GB table.

Follow-up: What does partitionBy on write control versus RDD partitions?

Lesson · Simulation

Short: Aim for ~100–256 MB per shuffle partition, then look at the histogram.

Detailed: Too few: fat tasks, spill, OOM. Too many: scheduler overhead. AQE coalesce helps small data. Skew means one partition will still be fat — salting, not more partitions, fixes that.

Senior: You want enough partitions for cores × duration, sized by data bytes, not by machine count.

Common mistake: Setting shuffle partitions to executor_count.

Follow-up: Why is executor_count the wrong formula?

Lesson · Simulation