Spark senior interview questions

Short: Task count and the slowest stage — not executor count.

Detailed: 80k tasks usually means tiny files or over-partitioning. Stage time is max(task). Find the stage with shuffle or scan skew, then file layout, then AQE/coalesce.

Senior: Hundreds to low thousands of healthy tasks, not tens of thousands of 50 ms tasks.

Common mistake: Doubling the cluster because 'there are many tasks'.

Follow-up: What number of tasks would you aim for on a 1 TB nightly?

Lesson · Simulation

Short: Analysis or optimization on the driver — missing table, bad types, or an exploding logical plan.

Detailed: If Spark UI has no job, the driver is still in Catalyst. Check analysis errors, huge SQL generation, or a view that unions 400 tables.

Common mistake: Looking at executor logs for a job that never submitted.

Follow-up: Where do you look if explain() itself hangs?

Lesson · Simulation

Short: Nothing was cached — cache is lazy until an action, or the plan changed so the cache key missed.

Detailed: Need an action after cache to materialize. If you then change the frame (extra filter), it is a new plan. Check Storage tab for the cached RDD/DataFrame.

Common mistake: Calling cache() and assuming the next cell is free.

Follow-up: When is checkpoint better than cache?

Lesson · Simulation

Short: Not always — both sides may need a shuffle for sort-merge. A broadcast join has zero Exchanges for the big side.

Detailed: SMJ typically shuffles both inputs unless already co-partitioned. BroadcastHashJoin shuffles neither of the fact side. Prove with explain().

Senior: Broadcast the small side, or bucket/co-partition both tables on the join key.

Common mistake: Removing 'an Exchange' without knowing join strategy.

Follow-up: How would you eliminate both shuffles?

Lesson · Simulation

Short: Shuffle fetch — a reducer could not read a map output, often because an executor died or shuffle service lost files.

Detailed: Not 'Spark is random'. Check executor loss, disk, and whether speculative execution is hiding a bad node. Retry may succeed; repeated FetchFailed means shuffle storage or OOM on the map side.

Senior: On the map-side executor local disk (or shuffle service). If that JVM dies hard, reducers fetch-fail.

Common mistake: Increasing retries without looking at executor logs.

Follow-up: Where are shuffle files stored?

Lesson · Simulation

Short: Each reducer partition is huge — spill, OOM, or hour-long tasks.

Detailed: 200 is a default, not a law. Target ~128–256 MB shuffle partitions. AQE coalesce helps small data; large data needs more partitions or a better plan (pre-agg, broadcast).

Senior: It can coalesce empty/small partitions after seeing map-side sizes — it will not save you from one skewed key.

Common mistake: Leaving 200 forever because 'that's Spark'.

Follow-up: How does AQE change this number at runtime?

Lesson · Simulation

Short: If input is tiny files or a full scan, the cluster is waiting on the lake, not the CPU.

Detailed: Check Spark UI: many tiny tasks, huge task overhead, or a scan that ignores partition filters. Scaling executors will not fix a thousand 2 MB files.

Senior: Fix the files or the plan. Then scale. Never the reverse as a first move.

Common mistake: Adding workers before looking at file sizes and partition pruning.

Follow-up: What would you change first — cluster size or the table layout?

Lesson · Simulation

Short: Driver OOM: notebook/kernel dies, executors still green. Executor OOM: one worker lost, driver alive, task failed with ExecutorLostFailure.

Detailed: Spark UI Executors tab: look at driver vs executor memory. Driver heap used near max plus a collect/broadcast in the plan → driver. One executor with huge spill or a killed container → executor.

Senior: collect, toPandas, and broadcasting a large dimension. Write the hypothesis, then the UI evidence.

Common mistake: Restarting the cluster without reading the error class.

Follow-up: Which action is the classic driver killer?

Lesson · Simulation

Short: Sometimes — RDD partitionBy with a custom Partitioner, or stay in DataFrame with salting.

Detailed: Prefer salting + AQE in DataFrame land. Custom partitioners are powerful and easy to get wrong (breaking join co-partitioning). Measure first.

Common mistake: Custom partitioner as the first skew fix.

Follow-up: What join property do you lose if only one side is custom-partitioned?

Lesson · Simulation

Short: The list lives on the driver; Spark then has to ship it.

Detailed: Parallelize/createDataFrame from driver memory. Read from storage or a Spark table instead. Same class of bug as collect() in reverse.

Common mistake: Raising executor memory for a driver-side Python list.

Follow-up: How should a 50 GB CSV enter Spark?

Lesson · Simulation

Short: Each withColumn can add projection overhead; UDFs in those columns serialize every row.

Detailed: Prefer select with expressions. If columns are Python UDFs, you lost Tungsten/Photon. Spark UI: time in Python vs CPU.

Common mistake: More executors for a UDF pipeline.

Follow-up: How do you rewrite 30 withColumns cleanly?

Lesson · Simulation

Short: When the notebook pulls a large unaggregated result to render.

Detailed: UI limits help, but a wide table with huge strings still hammers the driver. Prefer aggregations or LIMIT. Job clusters should not display fact tables.

Common mistake: Leaving display(df) in a scheduled notebook.

Follow-up: How do you QA a 2 TB write without pulling it?

Lesson · Simulation

Short: When invalid tables, bad schemas, or huge plans only fail at the action — in a job cluster at 2am.

Detailed: Validate schema and table existence early. Unit-test explain() on representative data. Don't leave 15 debug actions that each rerun a shuffle.

Common mistake: Adding count() everywhere as 'validation'.

Follow-up: What is a cheap eager check that isn't a full shuffle?

Lesson · Simulation

Short: Aim for ~100–256 MB per shuffle partition, then look at the histogram.

Detailed: Too few: fat tasks, spill, OOM. Too many: scheduler overhead. AQE coalesce helps small data. Skew means one partition will still be fat — salting, not more partitions, fixes that.

Senior: You want enough partitions for cores × duration, sized by data bytes, not by machine count.

Common mistake: Setting shuffle partitions to executor_count.

Follow-up: Why is executor_count the wrong formula?

Lesson · Simulation

Short: Broadcast the small side, or co-partition/bucket both sides on the join key.

Detailed: BroadcastHashJoin keeps the fact table in place. SortMergeJoin shuffles both unless already partitioned the same way. Prove with Exchange absence in explain().

Senior: When both sides are large and unsorted. Then make it balanced — not broadcast a 30 GB dimension.

Common mistake: spark.sql.shuffle.partitions as the join strategy.

Follow-up: When is a shuffle the right answer?

Lesson · Simulation

Short: It's a many-to-many — duplicates on the key, not Spark magic.

Detailed: Count distinct keys vs rows on both sides before the join. Look for duplicate dimension keys. Spark UI won't show 'duplicates'; data profiling will.

Senior: SELECT key, COUNT(*) FROM dim GROUP BY 1 HAVING COUNT(*) > 1 — then the same on the fact.

Common mistake: Raising shuffle partitions to fix a row explosion.

Follow-up: What's the SQL test you'd run in the interview whiteboard?

Lesson · Simulation

Short: Lost partitions recompute from lineage — or fail if checkpoint wasn't used and shuffle files are gone.

Detailed: Cache is not a durable store. Storage tab shows cached partitions per executor. Executor loss → recompute. That's why checkpoint or a Delta write is the durable form.

Senior: Storage tab + stage that skips the scan. If the scan is back, cache missed or was evicted.

Common mistake: Treating cache as a table.

Follow-up: How do you see cache hit vs recompute in the UI?

Lesson · Simulation

Short: Any action that pulls the full result to the driver — collect, toPandas, display of an unaggregated frame.

Detailed: Executor count does not grow driver heap. A 2 TB collect still targets one JVM. Broadcast join of a 8 GB dimension also hits the driver, then every executor.

Senior: Aggregate or sample on executors, write the rest, never pull the grain to the notebook.

Common mistake: Scaling workers to fix a driver OOM.

Follow-up: How would you rewrite the notebook so the driver stays small?

Lesson · Simulation

Short: 1 action → N stages (shuffles + 1) → M tasks per stage (partitions).

Detailed: Draw: show() → Job 1 → Stage 0 scan/filter → Exchange → Stage 1 aggregate → few rows to driver. Tasks = partitions in each stage, possibly different counts after shuffle.

Common mistake: Drawing one box called 'the cluster'.

Follow-up: What changes M without changing N?

Lesson · Simulation

Short: Find the join node and whether an Exchange sits above each child.

Detailed: No Exchange on the fact side → broadcast. Exchange on both → SMJ. That's the DAG, not the SQL text.

Common mistake: Reading only the SQL string.

Follow-up: Where is that visible besides explain()?

Lesson · Simulation

Short: ANALYZE TABLE / fix stats, add a broadcast hint if you know the size, or enable AQE join conversion.

Detailed: Don't hint forever as a substitute for stats. Hints rot. Measure with actual size after filters — AQE sees runtime sizes.

Senior: When you've measured the build side after filters and documented why stats cannot see it (UDF, JDBC, etc.).

Common mistake: Broadcast hint on a table that will grow 20×.

Follow-up: When is a hint the right senior answer?

Lesson · Simulation

Short: An operator that can't be compiled — often a UDF, an unsupported expression, or a fallback.

Detailed: explain() and SQL UI: look for the break in the * codegen star. Fix the expression. Don't 'tune Tungsten' as a config cult.

Common mistake: Random spark.tungsten flags from a blog.

Follow-up: How does Photon relate to Tungsten on Databricks?

Lesson · Simulation

Short: Skew: same task id always huge shuffle read for a key. Slow node: different keys, one host.

Detailed: Look at shuffle read bytes, not just duration. If bytes are even but duration isn't, suspect disk/CPU/noisy neighbor. If bytes are skewed, it's data.

Senior: Add salt, agg/join on (key, salt), then drop salt and re-agg if needed.

Common mistake: Speculative execution as the root-cause fix.

Follow-up: Walk through salting without ruining the grain.

Lesson · Simulation

Short: Driver collects the build side to ship the broadcast.

Detailed: If the dimension is 6 GB, the driver must hold it. Executors OOM later if they also can't hold the hashed relation — but the first failure can be the driver. Measure broadcast exchange size.

Senior: Error class → driver vs executor → action/broadcast in the plan → rewrite or size the JVM with a documented cap.

Common mistake: Only looking at executor logs.

Follow-up: What's your evidence-ordered debug?

Lesson · Simulation

Short: If one key is fat: salt or split. If all tasks are fat: more partitions or a smaller row (drop columns). Memory last.

Detailed: UI: shuffle read per task. One bar huge → skew. All bars huge → partition sizing or explode. GC thrash with even bytes → memory/config or too much cache.

Senior: Copy-pasting spark.memory.fraction from OSS blogs onto Photon/DBR without measuring.

Common mistake: A 2× memory ticket with no histogram attached.

Follow-up: What Spark config is a footgun with off-heap and Databricks?

Lesson · Simulation

Short: Yes if the build side was small after filters. Bad if stats were wrong and it now OOMs.

Detailed: AQE join conversion uses runtime size. Watch SQL adaptive UI. If it OOMs, disable conversion for that query or fix the filter so the build side is truly small.

Senior: Don't hint a slowly growing dim forever. Partition/filter it, or SMJ when it crosses a documented threshold.

Common mistake: Disabling AQE globally because one query failed.

Follow-up: How do you cap broadcast size in a lakehouse with growing dims?

Lesson · Simulation

Short: Coalesce packed leftover data into too few partitions, or skew was hidden.

Detailed: Coalesce is for empty/small partitions. If data is uneven, you can get fat tasks. Inspect post-AQE partition sizes. Disable coalesce for that query or salt.

Senior: spark.sql.adaptive.enabled and compare SQL UI plans — same query, different later stages.

Common mistake: Blaming AQE without looking at the adaptive plan.

Follow-up: How do you show AQE on/off in an interview demo?

Lesson · Simulation