Narrow vs wide transformations
UDF in a 'simple' filter
Symptoms
- Plan shows BatchEvalPython
- Photon disabled
Narrow vs wide transformations interview questions
Interactive Spark interview questions on Narrow vs wide transformations. Same topic as /learn/spark/transformations. Narrow transformations stay inside a partition. Wide ones shuffle. That difference is the stage boundary.
Question 1 of 3
beginner
Narrow vs wide transformationsWhat is a transformation versus an action?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is a transformation versus an action?
Narrow vs wide transformations · tap to open the answer
Short: Transformation: new DataFrame, lazy. Action: job, side effect or result.
Detailed: filter, select, groupBy, join are transformations. show, count, write, collect, take are actions. groupBy alone does not shuffle until an action.
Common mistake: Saying groupBy 'runs the shuffle immediately'.
Follow-up: Is cache a transformation or an action?
intermediateNarrow vs wide transformation — give one example of each and the cost.
Narrow vs wide transformations · tap to open the answer
Short: Narrow: filter/map — no shuffle. Wide: groupBy/join — shuffle and a new stage.
Detailed: Narrow tasks read only their partition. Wide tasks wait on Exchange. Broadcast join is wide in API but not a shuffle of the fact table.
Common mistake: Calling join always a shuffle join.
Follow-up: Why can a filter after a groupBy not reduce shuffle bytes of that groupBy?
seniorYou chained 30 withColumn calls. The job is slow but the plan looks simple. What happened?
Narrow vs wide transformations · tap to open the answer
Short: Each withColumn can add projection overhead; UDFs in those columns serialize every row.
Detailed: Prefer select with expressions. If columns are Python UDFs, you lost Tungsten/Photon. Spark UI: time in Python vs CPU.
Common mistake: More executors for a UDF pipeline.
Follow-up: How do you rewrite 30 withColumns cleanly?