Interview/Spark

Spark interview questions: Repartition vs coalesce vs partitionBy

Interactive Spark interview questions on Repartition vs coalesce vs partitionBy. Practice with the matching lesson. repartition shuffles for even partitions, coalesce only merges, and write.partitionBy is a different thing entirely.

Lesson · Simulation

repartition(200) and coalesce(200) both leave me with 200 partitions. What is actually different?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Repartition vs coalesce vs partitionBy

coalesce(1) before write turned a 40-executor cluster into one busy core

Symptoms

  • One executor working while 39 sit idle for hours
  • Job went from 8 minutes to 3 hours after 'we need a single CSV'
  • One task in the write stage — and in the stage feeding it

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

repartition(200) and coalesce(200) both leave me with 200 partitions. What is actually different?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: repartition does a full shuffle and evens out the data; coalesce merges existing partitions with no shuffle and keeps them uneven.

Why: repartition inserts an Exchange — round-robin, or hash if you pass columns — so partitions come out roughly equal in size. coalesce is a narrow operation that glues partitions together on the same executor, which is cheap but inherits whatever imbalance the input had.

Common mistake: Trying to use coalesce to increase the partition count — it can only reduce.

Follow-up they will ask: When is the uneven result of coalesce perfectly fine?

Lesson · Simulation

intermediate

Someone put coalesce(1) before the write to get a single output file, and now the entire pipeline runs single-threaded. Explain the mechanism.

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: coalesce has no shuffle boundary, so the reduced partition count pushes back and the upstream transformations run with one task too.

Why: Because coalesce does not create an Exchange, the join or aggregation feeding the write ends up inside the same stage at one partition — one task on one executor doing everything. repartition(1) adds an Exchange instead, so upstream work keeps its parallelism and only the final write is serial.

Common mistake: Believing coalesce only affects the write because that is where the call sits in the code.

Follow-up they will ask: So how do you actually produce one output file without wrecking the job?

Lesson · Simulation

senior

You have a job that joins on customer_id and then windows on customer_id. When is repartition('customer_id') worth paying for?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: When one shuffle on that key gets reused by more than one downstream operator.

Why: repartition('customer_id') hash-partitions rows so matching keys are co-located, and a following join or window on the same key can then skip its own Exchange. If only one operator needs the key, it would have shuffled once anyway — you have just paid for two shuffles instead of one.

Senior depth: explain('formatted') and count Exchange nodes before and after the change. If the count did not drop, the repartition bought you nothing.

Common mistake: Adding repartition(col) before every join out of habit and doubling the shuffle count.

Follow-up they will ask: How do you verify the second shuffle actually disappeared?

Lesson · Simulation

architect

spark.sql.shuffle.partitions is 200 and the code also calls repartition(1000). Which one wins, and where?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: They have different scopes: the config sizes the engine's own Exchanges, repartition(1000) sizes that one explicit shuffle.

Why: Every groupBy or sort-merge join Exchange targets spark.sql.shuffle.partitions unless AQE coalesces it at runtime; repartition(1000) is its own Exchange with its own target, and a later groupBy re-shuffles back to the configured number. df.write.partitionBy is a third, unrelated thing — it controls the on-disk directory layout, not in-memory partition count.

Senior depth: repartition on the same columns you pass to write.partitionBy, so each task owns one output directory and writes one decent-sized file instead of every task dropping a fragment into every directory.

Common mistake: Conflating write.partitionBy with repartition and expecting one to control the other.

Follow-up they will ask: How do you get a sane directory layout and reasonable file sizes at the same time?

Lesson · Simulation

beginner

A DataFrame has 200 partitions and you call coalesce(400). What do you end up with?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Still 200 — coalesce can only reduce, and asking for more is silently ignored.

Remember these

  • coalesce only reduces partition count; increasing it is a silent no-op
  • No Exchange appears in the plan, so there is no visual clue
  • repartition(n) is the only way to raise the count
  • Confirm with getNumPartitions rather than trusting the call

Why: coalesce merges existing partitions into fewer, larger ones inside the same stage; it has no mechanism to split a partition because splitting requires redistributing rows, and redistributing rows is a shuffle. So coalesce(400) on a 200-partition frame returns a frame with 200 partitions and raises no error, which is exactly why the bug survives code review. The plan shows no Exchange at all, so there is nothing in explain output to catch your eye either. If you genuinely want 400 partitions you must call repartition(400), which inserts Exchange RoundRobinPartitioning(400) and evens the sizes out.

print(df.rdd.getNumPartitions())        # 200
print(df.coalesce(400).rdd.getNumPartitions())   # 200
print(df.repartition(400).rdd.getNumPartitions()) # 400

Common mistake: Believing coalesce failed loudly somewhere, when it quietly returned the frame unchanged.

Follow-up they will ask: Why can repartition increase the count when coalesce cannot?

Lesson · Simulation

intermediate

You added repartition(200, 'customer_id') to fix a slow stage and one task is still carrying everything. Why did repartition not spread the data?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Hash partitioning on a key sends every row with that key to one partition, so a single hot key stays one fat task.

Remember these

  • repartition(n) is round-robin and even; repartition(n, col) is hash and co-located
  • One hot key always lands in exactly one hash partition
  • Raising n does not split a single key across partitions
  • Pass columns only when a downstream operator needs that key co-located

Why: repartition(n) with no columns is Exchange RoundRobinPartitioning, which distributes rows blindly and therefore evenly. repartition(n, col) is Exchange hashpartitioning(col, n), which guarantees co-location of equal keys, and co-location is the opposite of balance when one key dominates. If customer_id 0 or a sentinel like 'UNKNOWN' holds 40% of the rows, that 40% lands in exactly one of the 200 partitions no matter how large n gets. The Stage page shows it immediately: one task with enormous Shuffle Read and a large Spill (Disk) while the other 199 finish in seconds. The fix is salting the hot key, letting AQE skew handling split it, or dropping the key argument if you only wanted balance and never needed co-location.

df.repartition(200)                # even, no co-location
df.repartition(200, "customer_id") # co-located, skew preserved
df.groupBy("customer_id").count().orderBy(F.desc("count")).show(5)

Common mistake: Passing the join key to repartition when the real goal was balanced task sizes.

Follow-up they will ask: How would you confirm which key is hot without a full scan of the data?

Lesson · Simulation

advanced

In explain output you see Exchange RoundRobinPartitioning(200) in one plan and Exchange hashpartitioning(customer_id, 200) in another. What changes for the stage that follows?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Hash partitioning satisfies a downstream operator's distribution requirement, so the planner can drop its Exchange; round-robin cannot.

Remember these

  • EnsureRequirements removes an Exchange when the child partitioning already satisfies it
  • Hash partitioning on the key co-locates; round-robin only balances
  • Partition counts must match for the downstream Exchange to be dropped
  • Count Exchange nodes before and after to prove the optimization landed

Why: Every operator that needs co-location declares a required child distribution, and the EnsureRequirements rule inserts an Exchange only when the child's existing partitioning does not satisfy it. hashpartitioning(customer_id, 200) satisfies a following groupBy, SortMergeJoin, or Window on customer_id with the same partition count, so no second Exchange appears and you have paid for one shuffle instead of two. RoundRobinPartitioning satisfies nothing except 'some distribution', so the next wide operator shuffles again. The practical way to check is to count Exchange nodes in explain('formatted') before and after the change; if the count did not drop, the keyed repartition bought you nothing and you are now paying an extra shuffle for even-sized partitions you could have gotten for free. Round-robin also used to be a correctness hazard on retry because the row-to-partition assignment is not a function of the data, which is why Spark sorts within round-robin partitions before writing shuffle output.

df.repartition(200, "customer_id").groupBy("customer_id").sum("amount").explain("formatted")

Senior depth: The counts no longer match, so the groupBy's required distribution is not satisfied and the planner reinserts an Exchange at 400. You have paid for two shuffles and saved none. Either align the repartition count with the shuffle partition count or let the engine plan the single Exchange itself.

Common mistake: Assuming any repartition before a groupBy saves a shuffle, regardless of columns or count.

Follow-up they will ask: What happens if you repartition to 200 but spark.sql.shuffle.partitions is 400?

Lesson · Simulation

advanced

Your output volume swings ten-fold between a weekday and a sale day. repartition(64) gives 2 MB files on Tuesday and 2 GB files on Black Friday. How do you get stable file sizes?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Stop pinning a partition count and bound the write by rows or bytes per file instead.

Remember these

  • A fixed partition count fixes file count, not file size
  • maxRecordsPerFile rolls a new file within one write task
  • AQE coalescing handles the small-volume direction
  • Compaction is the backstop, not the primary control

Why: A fixed repartition count fixes the number of output files, which means file size is whatever the day's volume divided by that count happens to be. The writer-side control is maxRecordsPerFile, which makes a single write task roll to a new file once it has emitted that many rows, so a fat partition produces several right-sized files rather than one huge one. For the small-day direction, enable AQE coalescing so tiny post-shuffle partitions merge before the write instead of each becoming a fragment. On Delta, optimized writes and auto-compaction do this adaptively at the table level, and a scheduled OPTIMIZE fixes whatever still slips through. Compute the repartition count from an estimated byte volume if you must pick one, but treat the per-file cap as the thing that actually holds the contract.

(df.write
   .option("maxRecordsPerFile", 2_000_000)
   .mode("overwrite")
   .parquet("/mnt/curated/orders"))

Senior depth: Write one sample batch, divide output bytes by output rows to get an average compressed bytes-per-row, then divide your target file size by that. Re-check it after any schema change that adds wide columns, because bytes-per-row is the thing that drifts.

Common mistake: Hand-tuning the repartition count after every incident instead of bounding file size directly.

Follow-up they will ask: How would you pick the maxRecordsPerFile value for a table you have never written before?

Lesson · Simulation

senior

You filter a 2 TB table down to 3 GB and then call coalesce(50) before writing. The write looks fine but the job got much slower. What happened?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: coalesce has no shuffle boundary, so the 50 propagated back up into the scan and 50 tasks now read the full 2 TB.

Remember these

  • coalesce is narrow, so its partition count propagates up the whole narrow chain
  • The throttled stage shows huge Input Size with no shuffle metrics to blame
  • repartition(n) adds the Exchange that stops the propagation
  • The propagation stops only at the nearest upstream shuffle boundary

Why: coalesce creates a narrow dependency: each output partition is computed by one task that reads several parent partitions. Because there is no Exchange, there is no stage boundary either, so the filter and the scan feeding it are fused into the same stage as the write — and that stage has 50 tasks. Those 50 tasks are now responsible for scanning all 2 TB, not the 3 GB that survives the filter. The Spark UI shows one long stage with 50 tasks and a huge Input Size / Records, and nothing in the shuffle metrics to explain the time, which is exactly why this one is hard to spot. The fix is repartition(50), which inserts an Exchange so the scan keeps its original parallelism and only the post-shuffle write runs at 50.

big.filter("event_date = '2026-10-23'").coalesce(50).write.parquet(path)   # scan runs at 50
big.filter("event_date = '2026-10-23'").repartition(50).write.parquet(path) # scan keeps parallelism

Senior depth: When the stage it collapses into is already cheap — typically when the filter is a partition-pruning predicate so the scan reads little, or when the upstream work is already behind a shuffle boundary that coalesce cannot cross. Check where the nearest Exchange is before you decide; that boundary is the limit of the damage.

Common mistake: Reading coalesce as 'applies at the write' because that is where the call sits in the code.

Follow-up they will ask: When is coalesce after a filter actually the right call?

Lesson · Simulation