UDF in a 'simple' filter
Symptoms
- Plan shows BatchEvalPython
- Photon disabled
Interactive Spark interview questions on Narrow vs wide transformations. Practice with the matching lesson. Narrow transformations stay inside a partition. Wide ones shuffle. That difference is the stage boundary.
Question 1 of 8
What is a transformation versus an action?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 8 of 8 questions.
What is a transformation versus an action?
Narrow vs wide transformations · tap to open the answer
Say this first: Transformation: new DataFrame, lazy. Action: job, side effect or result.
Why: filter, select, groupBy, join are transformations. show, count, write, collect, take are actions. groupBy alone does not shuffle until an action.
Common mistake: Saying groupBy 'runs the shuffle immediately'.
Follow-up they will ask: Is cache a transformation or an action?
Narrow vs wide transformation — give one example of each and the cost.
Narrow vs wide transformations · tap to open the answer
Say this first: Narrow: filter/map — no shuffle. Wide: groupBy/join — shuffle and a new stage.
Why: Narrow tasks read only their partition. Wide tasks wait on Exchange. Broadcast join is wide in API but not a shuffle of the fact table.
Common mistake: Calling join always a shuffle join.
Follow-up they will ask: Why can a filter after a groupBy not reduce shuffle bytes of that groupBy?
You chained 30 withColumn calls. The job is slow but the plan looks simple. What happened?
Narrow vs wide transformations · tap to open the answer
Say this first: Each withColumn can add projection overhead; UDFs in those columns serialize every row.
Why: Prefer select with expressions. If columns are Python UDFs, you lost Tungsten/Photon. Spark UI: time in Python vs CPU.
Common mistake: More executors for a UDF pipeline.
Follow-up they will ask: How do you rewrite 30 withColumns cleanly?
df.drop('notes') versus df.select('id', 'amount') — which one changes what gets read from storage?
Narrow vs wide transformations · tap to open the answer
Say this first: Neither, directly. The optimizer derives the scan's required columns from the whole plan, so both end up pruning the same way.
Remember these
Why: Column pruning is a Catalyst rule, not an API behaviour: it walks the plan, collects the columns any operator actually needs, and pushes that set down into the file scan. Because Parquet and Delta are columnar, the reader then skips the unreferenced columns on disk entirely. So whether you expressed the intent as a drop or as a select, the FileScan's output schema is the same and you can read it right off explain('formatted'). What does defeat pruning is referencing the column after all — a UDF that takes the whole row, a select('*') late in the chain, or a write that happens to carry everything. The difference between drop and select is maintenance, not performance: select is an allow-list that fails when a required column disappears, drop is a deny-list that silently carries new upstream columns through.
df.drop('notes').explain('formatted')
# FileScan parquet [id#1,amount#2] -- notes never readCommon mistake: Claiming drop is wasteful because 'Spark reads the column first and then throws it away'.
Follow-up they will ask: Which one would you prefer in a pipeline whose source schema evolves?
Is df.orderBy('amount') narrow or wide, and why does it sometimes create an extra job?
Narrow vs wide transformations · tap to open the answer
Say this first: It is wide — a global sort range-partitions the data, and building the range boundaries requires sampling the input first.
Remember these
Why: A global ordering cannot be produced by sorting each partition independently, so Spark range-partitions: partition 0 gets the lowest values, partition 1 the next band, and so on, after which a local sort per partition yields a globally ordered result. To choose those band boundaries it has to know the value distribution, so it samples the input in a separate job before the shuffle — that is the short job you see appear ahead of the real work. The physical plan shows Exchange rangepartitioning followed by Sort, which is structurally different from the hashpartitioning Exchange a groupBy produces. Skew hurts a range partitioner directly: if one value dominates, its band becomes one enormous task, and the sort spills. When you only need ordering inside groups, sortWithinPartitions avoids the global shuffle entirely, and when you only need the top rows, orderBy().limit(n) is planned as TakeOrderedAndProject, which keeps a bounded heap per partition instead of sorting everything.
df.orderBy('amount').limit(10).explain() # TakeOrderedAndProject
df.sortWithinPartitions('amount') # no global shuffleCommon mistake: Assuming sort is narrow because each partition 'just sorts itself'.
Follow-up they will ask: Why is sortWithinPartitions enough before a window function in some cases?
distinct(), dropDuplicates(['id']), and groupBy('id').agg(max('ts')) all deduplicate. Pick one and defend it.
Narrow vs wide transformations · tap to open the answer
Say this first: distinct() dedupes on every column, dropDuplicates on a subset keeps an arbitrary row, and only an explicit aggregate or window makes the surviving row deterministic.
Remember these
Why: distinct() is shorthand for grouping by all columns, so it shuffles on the full row and plans as a HashAggregate with every column as a key — expensive and wrong if one column is a load timestamp that differs between duplicates. dropDuplicates(['id']) shuffles on id only and keeps whichever row arrives first within each partition, which means the winner can change between runs and after a retry. If which duplicate survives matters — and for late-arriving updates it always does — state the rule explicitly: groupBy the key and aggregate the fields, or use row_number() over a window partitioned by the key and ordered by the version column, then filter to rank one. The window form costs an Exchange plus a Sort and is the one to reach for when you need whole rows rather than a few aggregated fields. Whatever you choose, check the HashAggregate keys in the plan match the key you meant.
w = Window.partitionBy('id').orderBy(col('updated_at').desc())
dedup = df.withColumn('rn', row_number().over(w)).filter('rn = 1').drop('rn')Common mistake: Using dropDuplicates on a key and then being surprised that a rerun picks a different version of the row.
Follow-up they will ask: How would you dedupe deterministically without a window function?
If Catalyst pushes predicates anyway, does it matter whether you filter before or after a join?
Narrow vs wide transformations · tap to open the answer
Say this first: For inner joins it usually does not; for outer joins, aggregates, and windows the optimizer cannot push through, so placement changes both cost and results.
Remember these
Why: PushDownPredicate moves a filter below an inner join whenever the predicate references only one side, so the two spellings converge on the same physical plan. Outer joins break that: a filter on the null-producing side cannot be pushed without changing semantics, because the join is what manufactures those nulls — which is also why a where clause on the right side of a left outer join silently turns it into an inner join. Aggregates only let predicates through when they reference grouping keys, never aggregate outputs, which is the difference between WHERE and HAVING. Window functions are a hard barrier, since a filter above a window would change the frame the ranking was computed over. Python UDFs are opaque, so any predicate wrapping one stays where you put it. The practical rule is to filter as early as you can express it correctly and then verify with explain('formatted') that the predicate ended up in PushedFilters rather than in a Filter node above the Exchange.
# changes an outer join into an inner one
left.join(right, 'id', 'left').filter(right.status == 'OK')
# keeps outer semantics
left.join(right.filter("status = 'OK'"), 'id', 'left')Senior depth: The dangerous case is not the slow one, it is the one where pushdown is blocked for correctness and the author did not notice the semantics changed. I read every outer join's downstream filters in review, because that is the single most common silently-wrong pattern in DataFrame code.
Common mistake: Trusting the optimizer uniformly and shipping a left outer join that a later where clause quietly made inner.
Follow-up they will ask: How would you keep outer-join semantics while still filtering the right side early?
An explode on a nested events array takes down one executor every run. What is the mechanism and how do you bound it?
Narrow vs wide transformations · tap to open the answer
Say this first: explode multiplies rows inside the task that holds the parent row, so a single fat array produces a huge partition that no later repartition can rescue.
Remember these
Why: Generate — the plan node behind explode — is narrow: output rows are produced in the same task as their input row, with no opportunity to redistribute. So if one record carries an array of two million elements, one task must materialise all of it, and that task's memory pressure is what kills the executor, regardless of how many partitions the stage has overall. Adding repartition after the explode does not help, because the explosion has already happened by the time the shuffle sees the rows. The levers are all upstream. Measure size(col('events')) and look at the distribution before anything else, then increase input parallelism so each task holds fewer parent rows — a smaller spark.sql.files.maxPartitionBytes or an explicit repartition before the explode. Filter or cap pathological arrays if the business allows it, and if a few records are genuinely enormous, split them out and process that tail separately. The UI signature is specific: one task in a narrow stage with Peak Execution Memory far above the median, no shuffle involved.
df.select(size('events').alias('n')).summary('min', '75%', 'max').show()
df.repartition(4000).select(explode('events').alias('e'))Senior depth: Any row-multiplying operator needs its fan-out measured before it ships, the same way you would size a join. The failure is always in the tail of the array-length distribution, so the max matters far more than the average, and the memory ceiling is per-task — which is why this is one of the few problems a bigger cluster genuinely cannot solve.
Common mistake: Adding repartition immediately after the explode and reporting that Spark 'ignored' it.
Follow-up they will ask: How would you process the handful of records with million-element arrays?