Interview/Spark

Spark senior interview questions

Skew, shuffle, join strategy, Catalyst, and Spark UI evidence.

Lesson

A dashboard job has 1 job, 4 stages, 80k tasks. What do you inspect first?

Answer it out loud, then reveal. Play steps through like the simulators.

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 131 of 131 questions.

architect

A dashboard job has 1 job, 4 stages, 80k tasks. What do you inspect first?

What happens when Spark executes a job? · tap to open the answer

Say this first: Task count and the slowest stage — not executor count.

Why: 80k tasks usually means tiny files or over-partitioning. Stage time is max(task). Find the stage with shuffle or scan skew, then file layout, then AQE/coalesce.

Senior depth: Hundreds to low thousands of healthy tasks, not tens of thousands of 50 ms tasks.

Common mistake: Doubling the cluster because 'there are many tasks'.

Follow-up they will ask: What number of tasks would you aim for on a 1 TB nightly?

Lesson · Simulation

advanced

What does WholeStageCodegen actually fuse, and name two things that break the fusion.

What happens when Spark executes a job? · tap to open the answer

Say this first: It compiles a chain of supported operators into one generated Java method that loops over rows without virtual calls; unsupported operators split it into separate codegen stages.

Remember these

  • Codegen inlines an operator chain into one generated Java method
  • Fused operators carry an asterisk under WholeStageCodegen (n)
  • Python UDFs split the island via BatchEvalPython
  • maxFields and hugeMethodLimit cause silent fallback to interpreted mode

Why: Without codegen, each operator is an iterator calling next() on its child, so a simple filter-and-project pays a virtual call and an object materialisation per row per operator. Catalyst instead generates a single method whose body inlines the whole chain, so the row stays in registers and the JVM can JIT a tight loop; in explain('formatted') these appear as WholeStageCodegen (n) wrappers and the fused operators are marked with an asterisk. The fusion stops wherever an operator does not support codegen. A Python UDF is the clearest breaker: BatchEvalPython sits between two codegen islands because rows must leave the JVM entirely. Columnar sources break it at the boundary too, which is why you see ColumnarToRow between a vectorised FileScan and the first fused operator. Generated code also has limits — past spark.sql.codegen.maxFields or the bytecode size in spark.sql.codegen.hugeMethodLimit, Spark falls back to the interpreted path for that stage, so a 300-column select can quietly lose codegen while the plan still looks reasonable.

df.filter('amount > 100').select('id', 'amount').explain('formatted')
# look for '* Project' and 'WholeStageCodegen (1)'

Senior depth: Codegen is the reason expression-level work in the DataFrame API is nearly free and the reason a single UDF changes the cost structure of a whole stage. When I review a slow plan, I count codegen islands first: every extra island is a materialisation boundary somebody introduced, usually a UDF or an unnecessary conversion.

Common mistake: Reading the asterisk as 'this operator is fast' rather than 'this operator is inside a fused codegen stage'.

Follow-up they will ask: How would you detect codegen fallback on a very wide table?

Lesson · Simulation

advanced

A stage reports Spill (Disk) in the hundreds of gigabytes. Walk from the memory manager to that number.

What happens when Spark executes a job? · tap to open the answer

Say this first: Each task gets a slice of the executor's execution memory pool; when its sort or aggregation exceeds that slice, it writes sorted runs to local disk and merges them.

Remember these

  • Per-task budget is roughly the execution pool divided by cores
  • Sorts and hash aggregates spill sorted runs, then merge them
  • Spilled bytes are written and re-read, so disk dominates duration
  • Even spill across tasks means too few partitions; uneven means skew

Why: Spark's unified memory manager carves roughly spark.memory.fraction of the heap into a shared pool, split between execution and storage with spark.memory.storageFraction as the storage floor. Execution memory is then shared dynamically across the tasks running concurrently in that executor, so the effective budget per task is close to the execution pool divided by spark.executor.cores. A sort uses UnsafeExternalSorter and an aggregation uses a hash map that grows until its budget is exhausted; at that point the operator serialises what it has as a sorted run on local disk, resets, and continues, then merges the runs at the end. Every byte therefore gets written and read back at least once, which is why a spilling stage's duration is dominated by disk and not CPU. Two numbers tell you which lever to pull: if a few tasks spill enormously while the median spills nothing, it is skew; if every task spills evenly, the shuffle partitions are too few for the data volume. Raising spark.sql.shuffle.partitions or lowering cores per executor both increase per-task budget, from opposite directions.

spark.conf.set('spark.sql.shuffle.partitions', '2000')
# target roughly 128-256 MB per shuffle partition

Senior depth: Spill is not a failure, it is the engine degrading gracefully — the question is whether you are paying for it on purpose. I compare Spill (Disk) against Shuffle Read per task before touching any config, because that ratio distinguishes 'partition slightly too big' from 'one key holds the stage'.

Common mistake: Raising spark.executor.memory while leaving spark.executor.cores high, so the per-task share barely moves.

Follow-up they will ask: Why does lowering spark.executor.cores sometimes make a spilling stage faster?

Lesson · Simulation

senior

A stage was retried after a failure, the job succeeded, and the output row count differs from the previous run. How can a retry change results?

What happens when Spark executes a job? · tap to open the answer

Say this first: If a stage's output is not deterministic, recomputing one partition can produce a different split of rows than the partitions downstream already consumed.

Remember these

  • Recovery assumes a partition recomputes to identical contents
  • repartition() without a key, unseeded rand(), and generated ids break that
  • Indeterminate stages force a full stage and downstream rerun
  • Already-emitted external side effects cannot be rolled back

Why: Spark's recovery model assumes a lost partition can be recomputed identically from its lineage. That assumption breaks when the upstream output order is not stable — repartition() without a key, rand() without a seed, monotonically_increasing_id(), or a source that lists files in a nondeterministic order. If a reducer fails and Spark recomputes only its input partition, rows can land in a different partition than they did the first time, so some rows are processed twice and others not at all. Spark 3 addresses this by marking such stages indeterminate: when one partition of an indeterminate stage must be recomputed, the scheduler invalidates and reruns the whole stage and everything downstream, which is correct but expensive and shows up as a surprising wave of retried stages in the UI. What it cannot fix is work that already escaped the engine, such as a non-idempotent external write inside foreachPartition. So treat 'succeeded after retries' plus 'row count moved' as a determinism bug, not as flaky infrastructure. Use repartition(col) or a seeded rand(), and make external writes idempotent.

df.repartition(200, 'customer_id')   # deterministic placement
df.withColumn('r', rand(seed=42))    # reproducible across retries

Senior depth: Determinism is a correctness property of the pipeline, not a Spark setting. In review I look for unkeyed repartition, unseeded randomness, and generated surrogate keys before I look at performance, because those are the bugs that only appear on the night a node dies and then get closed as 'transient'.

Common mistake: Writing it off as a flaky node because the job eventually reported success.

Follow-up they will ask: Which of those nondeterminism sources is safe to keep if you add a checkpoint?

Lesson · Simulation

senior

The plan is huge but the job never starts. What failed?

What happens when Spark executes a job? · tap to open the answer

Say this first: Analysis or optimization on the driver — missing table, bad types, or an exploding logical plan.

Why: If Spark UI has no job, the driver is still in Catalyst. Check analysis errors, huge SQL generation, or a view that unions 400 tables.

Common mistake: Looking at executor logs for a job that never submitted.

Follow-up they will ask: Where do you look if explain() itself hangs?

Lesson · Simulation

senior

Two actions on the same cached DataFrame still scan twice. Why?

What happens when Spark executes a job? · tap to open the answer

Say this first: Nothing was cached — cache is lazy until an action, or the plan changed so the cache key missed.

Why: Need an action after cache to materialize. If you then change the frame (extra filter), it is a new plan. Check Storage tab for the cached RDD/DataFrame.

Common mistake: Calling cache() and assuming the next cell is free.

Follow-up they will ask: When is checkpoint better than cache?

Lesson · Simulation

senior

You see two Exchanges for one join. Is that always wrong?

What happens when Spark executes a job? · tap to open the answer

Say this first: Not always — both sides may need a shuffle for sort-merge. A broadcast join has zero Exchanges for the big side.

Why: SMJ typically shuffles both inputs unless already co-partitioned. BroadcastHashJoin shuffles neither of the fact side. Prove with explain().

Senior depth: Broadcast the small side, or bucket/co-partition both tables on the join key.

Common mistake: Removing 'an Exchange' without knowing join strategy.

Follow-up they will ask: How would you eliminate both shuffles?

Lesson · Simulation

senior

Tasks fail with FetchFailed. What layer actually broke?

What happens when Spark executes a job? · tap to open the answer

Say this first: Shuffle fetch — a reducer could not read a map output, often because an executor died or shuffle service lost files.

Why: Not 'Spark is random'. Check executor loss, disk, and whether speculative execution is hiding a bad node. Retry may succeed; repeated FetchFailed means shuffle storage or OOM on the map side.

Senior depth: On the map-side executor local disk (or shuffle service). If that JVM dies hard, reducers fetch-fail.

Common mistake: Increasing retries without looking at executor logs.

Follow-up they will ask: Where are shuffle files stored?

Lesson · Simulation

senior

spark.sql.shuffle.partitions = 200 on a 4 TB agg. What goes wrong?

What happens when Spark executes a job? · tap to open the answer

Say this first: Each reducer partition is huge — spill, OOM, or hour-long tasks.

Why: 200 is a default, not a law. Target ~128–256 MB shuffle partitions. AQE coalesce helps small data; large data needs more partitions or a better plan (pre-agg, broadcast).

Senior depth: It can coalesce empty/small partitions after seeing map-side sizes — it will not save you from one skewed key.

Common mistake: Leaving 200 forever because 'that's Spark'.

Follow-up they will ask: How does AQE change this number at runtime?

Lesson · Simulation

senior

A job is 'Spark' but the SLA miss is storage. How do you tell?

What is Apache Spark? · tap to open the answer

Say this first: If input is tiny files or a full scan, the cluster is waiting on the lake, not the CPU.

Why: Check Spark UI: many tiny tasks, huge task overhead, or a scan that ignores partition filters. Scaling executors will not fix a thousand 2 MB files.

Senior depth: Fix the files or the plan. Then scale. Never the reverse as a first move.

Common mistake: Adding workers before looking at file sizes and partition pruning.

Follow-up they will ask: What would you change first — cluster size or the table layout?

Lesson · Simulation

advanced

You run spark.read.parquet(path) with no action afterwards, and a job still appears in the Jobs tab. Explain it.

What is Apache Spark? · tap to open the answer

Say this first: Reading Parquet needs a schema and a file list, and both are resolved eagerly on the driver — the listing itself can be parallelised into a real job.

Remember these

  • Schema resolution happens at read time, not at action time
  • Large path counts trigger a distributed listing job
  • mergeSchema reads every footer instead of one
  • An explicit schema or a Delta log skips the listing cost

Why: A DataFrame needs a resolved schema before Catalyst can analyse anything, so the Parquet source reads at least one file footer at read time rather than at action time. Path discovery is also eager: the driver lists the directory, and once the number of input paths crosses spark.sql.sources.parallelPartitionDiscovery.threshold, Spark distributes the listing to executors, which shows up as a short job usually labelled with 'parquet at NativeMethodAccessorImpl' or 'InMemoryFileIndex'. On object storage this is the stage that hurts, because listing a prefix with hundreds of thousands of keys is many paginated API calls, not a directory read. If you set spark.sql.parquet.mergeSchema, Spark reads every footer instead of one, which turns a metadata peek into a full-table metadata scan. Supplying an explicit schema, or reading a Delta table whose transaction log already holds the file list and the schema, removes this entirely. So 'transformations are lazy' is true about row processing and false about schema and path resolution.

schema = spark.table('bronze.orders').schema
df = spark.read.schema(schema).parquet('s3://lake/orders')
# no footer read, no listing job

Senior depth: Treat file listing as a first-class cost line on object storage. Delta and Iceberg exist partly because listing is the scaling wall, not the scan. If you must stay on plain Parquet, keep directories shallow, partition coarsely, and never leave mergeSchema on in a scheduled job.

Common mistake: Insisting nothing touches storage until the first action, then having no explanation for a job that appears before one.

Follow-up they will ask: Which file format removes the listing cost entirely, and why?

Lesson · Simulation

senior

The cluster has been up 25 minutes and the Jobs tab is still empty. Give three different explanations and how you would separate them.

What is Apache Spark? · tap to open the answer

Say this first: Either no action was called, the driver is still in planning or file listing, or no executors were ever granted.

Remember these

  • Executors tab showing only the driver means allocation failed
  • Idle executors plus a busy driver means planning or listing
  • Driver thread dump names the blocking stack immediately
  • Empty Jobs tab with healthy everything often means no action

Why: Start with the Executors tab: if the only row is the driver, the cluster manager never gave you containers, and the problem is quota, instance availability, or an init script failing on every worker — not Spark. If executors are registered and idle, the driver is doing driver-only work: catalog resolution, a view that unions hundreds of tables, or an object-store listing against a huge prefix. A driver thread dump settles that in one click, because you will see threads parked in InMemoryFileIndex, the metastore client, or Catalyst rule application. The third case is the boring one: the code built a plan and never called an action, so there is genuinely nothing to run; explain() confirms a valid plan with zero jobs. Each branch has a different owner, which is the point of asking — platform, data layout, or the notebook. Checking executor logs first is wasted time in two of the three cases.

Senior depth: Order the checks by blast radius, not by familiarity: allocation, then driver, then code. Driver thread dump before log grep is the habit that separates people who fix this in two minutes from people who restart the cluster and hope.

Common mistake: Tailing executor stderr for a query that has not submitted a single task.

Follow-up they will ask: Which of those three would autoscaling make worse?

Lesson · Simulation

architect

You inherit a platform where every pipeline is Spark, including dozens that process under a gigabyte. What do you change, and how do you justify it?

What is Apache Spark? · tap to open the answer

Say this first: Route small, predictable jobs off Spark or onto shared serverless compute, and keep Spark for jobs whose data or fan-out actually needs a cluster.

Remember these

  • Sub-GB jobs are dominated by startup and scheduling, not work
  • Tuning shuffle partitions and shared compute beats rewriting first
  • Keep small jobs in Spark when they sit inside a larger lineage
  • Write the threshold down so it is not relitigated per team

Why: A sub-gigabyte job on a dedicated cluster pays cluster startup, a driver JVM, and a shuffle whose partition count was never tuned, so the wall clock is dominated by fixed cost rather than work; the Stages page shows thousands of tasks finishing in tens of milliseconds with visible Scheduler Delay. The cheap first move is not a rewrite: set a sane spark.sql.shuffle.partitions per job, leave AQE coalescing on so small post-shuffle partitions merge, and move these jobs onto shared or serverless compute so they stop paying individual cluster spin-up. Then classify by data volume and dependency shape. Jobs that are small and self-contained belong in a warehouse query or a single-container script; jobs that are small but sit in the middle of a Spark DAG should stay in Spark so you do not fragment the lineage. Publish the rule as a threshold in the platform docs rather than deciding case by case, otherwise every team relitigates it. The honest cost is two execution stacks to operate instead of one.

Senior depth: Instrument it: ratio of cluster-up time to task time per job. Anything above roughly one is paying more for compute existence than for compute. That single ratio turns a taste argument into a backlog you can rank.

Common mistake: Proposing a mass rewrite off Spark before measuring how much of the runtime is cluster startup.

Follow-up they will ask: What metric would you put on a dashboard to find these jobs automatically?

Lesson · Simulation

senior

How do you prove a failure is driver-side vs executor-side in five minutes?

Spark cluster architecture · tap to open the answer

Say this first: Driver OOM: notebook/kernel dies, executors still green. Executor OOM: one worker lost, driver alive, task failed with ExecutorLostFailure.

Why: Spark UI Executors tab: look at driver vs executor memory. Driver heap used near max plus a collect/broadcast in the plan → driver. One executor with huge spill or a killed container → executor.

Senior depth: collect, toPandas, and broadcasting a large dimension. Write the hypothesis, then the UI evidence.

Common mistake: Restarting the cluster without reading the error class.

Follow-up they will ask: Which action is the classic driver killer?

Lesson · Simulation

advanced

An executor container was killed with exit code 137, but the Peak JVM Memory graph never came close to the limit. How is that possible?

Spark cluster architecture · tap to open the answer

Say this first: The container limit covers JVM heap plus overhead plus off-heap plus Python, and only the first of those shows up as heap usage.

Remember these

  • Container limit = spark.executor.memory + spark.executor.memoryOverhead
  • Overhead holds metaspace, thread stacks, netty shuffle buffers, off-heap
  • PySpark spawns one python worker per concurrent task, outside the heap
  • Exit 137 is a kernel kill; java.lang.OutOfMemoryError is a heap kill

Why: The cluster manager enforces a limit on total resident memory for the container, which is roughly spark.executor.memory plus spark.executor.memoryOverhead. Overhead is not slack: it holds JVM metaspace, thread stacks, the netty buffers used for shuffle fetches, and any off-heap Tungsten region you enabled with spark.memory.offHeap.enabled and spark.memory.offHeap.size. With PySpark it also has to cover the python worker processes, one per concurrent task, which is why a job that was healthy in Scala starts getting OOM-killed the week someone adds a Python UDF. Heap graphs look fine because none of that growth is heap — the kernel kills the container while the JVM believes it has room, so you get exit 137 or 'Container killed by YARN for exceeding memory limits' rather than java.lang.OutOfMemoryError. The error class is the diagnostic: heap OOM means tune heap or reduce per-task data; 137 means raise overhead or lower concurrency per executor.

spark.conf.set('spark.executor.memoryOverhead', '4g')
spark.conf.set('spark.executor.pyspark.memory', '2g')

Senior depth: Read the error class before touching a number. Heap OOM and container kill point in opposite directions, and the single most common wasted day on a Spark platform is raising executor memory against a 137. Fewer cores per executor is often the cheaper fix because it directly cuts the number of concurrent python workers and shuffle buffers per container.

Common mistake: Raising spark.executor.memory in response to exit 137, which shrinks the overhead headroom and makes the kill arrive sooner.

Follow-up they will ask: Why can lowering spark.executor.cores fix an exit-137 loop without adding memory?

Lesson · Simulation

senior

Why does Spark usually prefer several mid-sized executors per node over one executor that owns the whole machine, and where does that advice break?

Spark cluster architecture · tap to open the answer

Say this first: Mid-sized executors keep GC pauses and I/O concurrency manageable; too many of them multiply broadcast copies and shuffle connections.

Remember these

  • One giant heap means cluster-wide stalls on full GC
  • High spark.executor.cores shrinks each task's execution memory share
  • Broadcasts are materialised once per executor, not per core
  • More executors means more shuffle locations and fetch connections

Why: A single huge executor means one JVM heap of tens of gigabytes, and full GC on that heap stalls every task in the container at once, which you see as a GC Time column rivalling task duration. Very high spark.executor.cores also means many tasks sharing one execution memory pool, so each task's share shrinks and sorts start spilling earlier. The usual landing zone is a handful of cores per executor, because object-store and HDFS client throughput stops improving past that point anyway. The opposite extreme is just as bad: one core per executor pays full JVM startup and metaspace per task slot, and every broadcast variable is materialised once per executor, so a 500 MB broadcast across 200 tiny executors is 100 GB of duplicated memory cluster-wide. More executors also means more shuffle map-output locations and more concurrent fetch connections per reducer, which is what turns a marginal network into FetchFailed. Size for the job shape: shuffle-heavy wants memory per task, scan-heavy wants cores.

Senior depth: The real variable is memory per task slot, not memory per executor. Divide the usable execution pool by cores and compare it against the largest shuffle partition you expect. If that number is below the partition size you are choosing to spill, and no amount of cluster growth changes it.

Common mistake: Quoting 'five cores per executor' as a law without mentioning broadcast duplication or per-task memory share.

Follow-up they will ask: How does dynamic allocation change how you pick executor size?

Lesson · Simulation

architect

Thirty teams have to share one Spark platform. Where do you put the isolation boundaries?

Spark cluster architecture · tap to open the answer

Say this first: Separate Spark applications per team workload, job clusters for scheduled work, and a shared-but-governed tier only for interactive exploration.

Remember these

  • Within one application there is no memory or driver isolation
  • FAIR scheduler pools share task slots, not memory or fault domains
  • Job clusters make the Spark application the fault boundary
  • Compute sharing must not imply data access; grants live in the catalog

Why: Inside one Spark application, isolation is weak: all notebooks share the same executors, the same block manager, and the same driver, so one person's collect() can kill everyone's session. Scheduler pools with spark.scheduler.mode set to FAIR divide task slots between concurrent queries, which smooths latency but does nothing about memory or a driver crash. That makes the application the only real fault boundary, so scheduled pipelines each get their own job cluster that lives for the duration of the run and dies with it. Interactive work is the hard case, because per-user clusters are expensive and idle most of the day; a shared cluster with autoscaling and enforced timeouts plus a serverless warehouse for SQL covers most of it. On top of compute, the data boundary is the catalog: table-level grants and separate schemas per domain, so compute sharing never implies read access. Document which tier is allowed to run production writes, or the shared tier quietly becomes production.

Senior depth: Pick the boundary by failure mode, not by cost. The question is always 'who else goes down with this', and the answer inside a single Spark application is 'everyone on the driver'. Cost optimisation on shared clusters is worth doing, but only after production writes are off them.

Common mistake: Treating FAIR scheduler pools as isolation and then being surprised when one driver OOM takes down thirty users.

Follow-up they will ask: What would you measure to decide whether the shared interactive tier is too small?

Lesson · Simulation

senior

You need a custom partitioner for a skewed key. Do you drop to RDD?

RDDs: lineage and partitions · tap to open the answer

Say this first: Sometimes — RDD partitionBy with a custom Partitioner, or stay in DataFrame with salting.

Why: Prefer salting + AQE in DataFrame land. Custom partitioners are powerful and easy to get wrong (breaking join co-partitioning). Measure first.

Common mistake: Custom partitioner as the first skew fix.

Follow-up they will ask: What join property do you lose if only one side is custom-partitioned?

Lesson · Simulation

advanced

What does mapPartitions give you over map, and what is the bug people hit with it?

RDDs: lineage and partitions · tap to open the answer

Say this first: It hands you an iterator per partition so you can amortise setup cost — and the bug is materialising that iterator into a list.

Remember these

  • mapPartitions runs setup once per task, not once per row
  • list(rows) materialises the whole partition and reintroduces OOM
  • Return a generator or yield so rows stay streamed
  • Resources must stay open until the lazy iterator is consumed

Why: map invokes your function once per record, so anything expensive to construct — a database connection, a tokenizer, a loaded model — is built per row. mapPartitions invokes it once per partition with an iterator over that partition's records, so the setup happens once per task and the teardown can be explicit. The trap is that an iterator is lazy and a list is not: writing list(rows) at the top of the function pulls the whole partition into memory at once, which turns Spark's streaming execution into a per-task memory ceiling and produces exactly the OOM that mapPartitions was supposed to avoid. Return a generator, or yield, so rows stay streamed. Two related mistakes are closing a resource in a way that triggers before the lazy iterator is consumed, and capturing a non-serialisable object in the closure instead of creating it inside the function. The payoff is real when setup is expensive; for pure column arithmetic, a DataFrame expression is still better than either.

def f(rows):
    model = load_model()      # once per partition
    for r in rows:
        yield (r.id, model.score(r))

rdd.mapPartitions(f)

Senior depth: mapPartitions is the right escape hatch exactly when per-partition setup dominates per-row work, and partition count then becomes a tuning knob against setup cost. The failure mode is always memory, because the API hands you something that looks like a collection and is not — if the function body does not read like a stream, it is wrong.

Common mistake: Writing rows = list(rows) for convenience and then reporting that mapPartitions causes executor OOM.

Follow-up they will ask: How would you size partitions for a function whose setup costs two seconds?

Lesson · Simulation

advanced

You wrapped the job in a class and rdd.map now fails with a pickling or 'Task not serializable' error. Explain the mechanism.

RDDs: lineage and partitions · tap to open the answer

Say this first: The closure captured self, so Spark has to serialise the whole enclosing object — including whatever non-serialisable field it holds.

Remember these

  • Referencing self.x in a closure captures the entire enclosing object
  • Bind values to locals so only the value is serialised
  • Broadcast large read-only payloads; build connections in mapPartitions
  • spark.serializer affects data serialisation, not closure capture

Why: A lambda that references self.multiplier does not capture a number; it captures self, and Spark must ship that object to every executor along with the task. If the class holds a SparkSession, a database connection, a logger, or a thread pool, serialisation fails, and in PySpark the message is a PicklingError naming the offending attribute rather than the line you think is at fault. The direct fix is to bind what you need to a local variable before the closure so only that value is captured. For anything large, use a broadcast variable so the payload is shipped once per executor instead of once per task, and for per-partition resources construct them inside mapPartitions so they are never serialised at all. A useful rule is that executor-bound code should be a module-level function or a static method, with configuration passed in as plain values. Kryo via spark.serializer affects how data is serialised, not how closures are captured, so switching it does not fix this class of error.

m = self.multiplier          # capture the value, not self
rdd.map(lambda r: r * m)
b = sc.broadcast(self.big_lookup)  # one copy per executor

Senior depth: Closure capture is the boundary between driver code and executor code, and it is invisible in Python until it fails. I write executor functions as module-level functions taking plain arguments, which makes the boundary explicit and makes the same functions unit-testable without a cluster.

Common mistake: Switching to KryoSerializer to fix a closure-capture error it has nothing to do with.

Follow-up they will ask: When is a broadcast variable the right answer rather than a local capture?

Lesson · Simulation

senior

A legacy source has no DataSource connector and three thousand shards. How do RDDs help, and what do you give up?

RDDs: lineage and partitions · tap to open the answer

Say this first: Parallelise the shard identifiers and fetch inside mapPartitions, then convert to a DataFrame as early as possible.

Remember these

  • Parallelise shard ids, not data; fetch inside mapPartitions
  • Set partition count from executor cores and source rate limits
  • Convert to a DataFrame with an explicit schema immediately
  • You own pushdown, statistics, and retry semantics at that boundary

Why: The pattern is to treat the shard list as the dataset: parallelise the three thousand identifiers into an RDD with a partition count matched to your executor cores and the source's rate limits, then use mapPartitions so one connection serves many shards and rows are yielded as a stream. That gives you exactly what you need from the RDD API — explicit control over parallelism and per-partition resource lifecycle — and nothing more. Convert the result to a DataFrame with an explicit schema immediately, so every downstream operation is back under Catalyst with codegen, pruning, and Photon eligibility. What you give up at the source boundary is pushdown and statistics: the engine cannot ask the legacy system to filter, so any selectivity has to be expressed in the fetch logic by hand. You also own retry semantics, because a failed task re-fetches its shards and the source may not be idempotent, which argues for landing raw into a bronze Delta table and doing all real processing from there. Keep the RDD island as small as you can defend.

shards = spark.sparkContext.parallelize(shard_ids, 200)
rows = shards.mapPartitions(fetch_shards)
bronze = spark.createDataFrame(rows, schema).write.saveAsTable('bronze.legacy')

Senior depth: The discipline is to land raw and process structured. An RDD ingestion island writing straight into bronze Delta keeps the unoptimisable part of the pipeline one stage deep, makes re-processing free, and means a source outage never forces you to re-run the transformations.

Common mistake: Keeping the whole pipeline in RDDs because the ingestion step had to be, instead of converting right after the fetch.

Follow-up they will ask: How would you make the fetch safe against task retries?

Lesson · Simulation

senior

createDataFrame(huge_python_list) OOMs before any executor works. Why?

DataFrames and schemas · tap to open the answer

Say this first: The list lives on the driver; Spark then has to ship it.

Why: Parallelize/createDataFrame from driver memory. Read from storage or a Spark table instead. Same class of bug as collect() in reverse.

Common mistake: Raising executor memory for a driver-side Python list.

Follow-up they will ask: How should a 50 GB CSV enter Spark?

Lesson · Simulation

advanced

After a currency conversion and a sum, a decimal revenue column comes back as nulls. What is the mechanism?

DataFrames and schemas · tap to open the answer

Say this first: Decimal arithmetic promotes precision and scale, and when the result needs more precision than decimal's 38-digit maximum, Spark returns null instead of raising by default.

Remember these

  • Decimal result types are derived, and precision is capped at 38 digits
  • allowPrecisionLoss trims scale instead of failing
  • Non-representable results become null unless ANSI mode is on
  • Only the largest magnitudes go null, which makes it look like bad data

Why: Every decimal operation derives a result type from its inputs: multiplication adds the precisions and the scales, and division inflates scale further. Chain a few operations on decimal(38,18) values and the derived type exceeds the 38-digit limit, so Spark has to reduce it. With spark.sql.decimalOperations.allowPrecisionLoss enabled, it keeps precision at 38 and trims scale, which quietly rounds; when a value genuinely cannot be represented, the result becomes null rather than an error unless spark.sql.ansi.enabled is on, in which case you get an overflow exception instead. That is why a revenue column turns to nulls after a step that looks arithmetically harmless, and why the nulls appear only for the largest values. The diagnosis is to print the schema after each step and watch the decimal type inflate. The fix is to control the type deliberately: cast to a scale you actually need before multiplying, or use double where you are summing approximate money-free quantities.

df.select((col('amt') * col('fx')).alias('rev')).printSchema()
df.select(col('amt').cast('decimal(20,4)') * col('fx').cast('decimal(12,6)'))

Senior depth: Silent null on overflow is the worst default for financial data because it is indistinguishable from a missing value. On money pipelines I pin explicit decimal types at each boundary and run with ANSI mode so overflow is an incident instead of a slowly wrong dashboard.

Common mistake: Blaming the upstream source for nulls that Spark's own decimal promotion produced.

Follow-up they will ask: What does turning on spark.sql.ansi.enabled change about this failure?

Lesson · Simulation

senior

Your Parquet files declare a column as required, Spark reads it as nullable, and a downstream assertion breaks. Why does Spark do that?

DataFrames and schemas · tap to open the answer

Say this first: Spark treats nullability as an optimizer hint rather than an enforced constraint, and readers widen to nullable to stay safe across files.

Remember these

  • nullable is a codegen hint, not an enforced constraint
  • File sources widen to nullable because a scan spans many files
  • Outer joins and aggregates legitimately widen nullability
  • Enforce non-null with Delta constraints, validated on write

Why: The nullable flag on a StructField tells Catalyst it may skip null handling in generated code — it is not a validated constraint on the data. Because a scan can span many files with slightly different writers, the Parquet source generally reports columns as nullable so that one file with nulls cannot produce wrong results or a crash in codegen generated under the opposite assumption. Operations also legitimately widen nullability: the right side of a left outer join becomes nullable, and so does any column under a window or an aggregate that can see empty groups. If you force nullable=False on a schema that actually contains nulls, you are lying to the optimizer, and the symptom is not a clean error but wrong results or a NullPointerException deep in generated code. The place to enforce 'this is never null' is the storage layer — a Delta NOT NULL constraint or a CHECK constraint is validated on write, every write, by every writer. Assert on data with a count of nulls, not on the schema flag.

spark.sql("ALTER TABLE silver.orders ALTER COLUMN order_id SET NOT NULL")
df.filter(col('order_id').isNull()).count()  # assert on data, not on the flag

Senior depth: Schema flags describe what the optimizer is allowed to assume; constraints describe what the data is allowed to be. Teams that conflate the two end up with schemas that assert guarantees nothing validates, and the failure surfaces as a codegen NullPointerException rather than a data quality alert.

Common mistake: Forcing nullable=False in a hand-built StructType to make a downstream check pass.

Follow-up they will ask: Which join type changes nullability on which side?

Lesson · Simulation

senior

A managed table and an external table both show up in SHOW TABLES. What changes the moment someone runs DROP TABLE?

Spark SQL: the same engine, a different keyboard · tap to open the answer

Say this first: Dropping a managed table deletes the data files; dropping an external table only removes the metadata.

Why: A managed table's location is owned by the catalog, so DROP removes the metastore entry and the underlying directory. An external table keeps its files at the LOCATION you declared — DROP leaves the data and you can re-register it. DESCRIBE EXTENDED shows Type: MANAGED vs EXTERNAL along with the Location.

Senior depth: DESCRIBE EXTENDED before any DROP in a runbook. Managed is fine for data the pipeline owns; external is for data other systems also write.

Common mistake: Saying DROP TABLE never deletes data because 'Spark doesn't own storage'.

Follow-up they will ask: How do you verify which kind you are about to drop?

Lesson · Simulation

architect

A scheduled SQL pipeline fails with an AnalysisException and Spark UI shows zero jobs for that run. Where do you look, and what is it definitely not?

Spark SQL: the same engine, a different keyboard · tap to open the answer

Say this first: It failed in Catalyst on the driver — name and type resolution — so it is not an executor or data-volume problem.

Why: Analysis resolves tables, columns, and functions against the catalog before a single stage is submitted, so a resolution failure means no job, no stage, and nothing in executor logs. spark.sql() builds and analyzes its plan when called, so the guilty statement is the first one referencing the renamed column or dropped table — not the write at the bottom of the notebook.

Senior depth: File permissions, corrupt Parquet footers, and Delta schema-on-write mismatches — those need a task to run. Resolution failures never get that far.

Common mistake: Digging through executor stderr for a query that never submitted a task.

Follow-up they will ask: Which failures do get deferred until execution time?

Lesson · Simulation

advanced

A filter on a partitioned table still read every partition. Give three reasons pruning can fail.

Spark SQL: the same engine, a different keyboard · tap to open the answer

Say this first: A function or cast applied to the partition column, a predicate on a non-partition column, or a join-derived filter that needs dynamic partition pruning.

Remember these

  • Pruning needs the bare partition column, with no cast or function
  • PartitionFilters empty plus DataFilters populated is the tell
  • Join-derived filters require dynamic partition pruning to work
  • DPP depends on the dimension side staying broadcastable

Why: Static pruning requires the optimizer to evaluate the predicate against partition values before the scan, so it must see the bare partition column on one side. Wrapping it defeats that: if the table is partitioned by a string event_date and you write where year(event_date) = 2024, or where cast(event_date as date) >= '2024-01-01', the predicate lands in DataFilters rather than PartitionFilters and every directory is read. The second case is simply the wrong column — filtering on a high-cardinality data column prunes nothing at the directory level, though Parquet row-group statistics or Delta data skipping may still help inside the files. The third case is a filter that exists only through a join, as when a date dimension restricts the fact table; that needs spark.sql.optimizer.dynamicPartitionPruning.enabled, and it only applies when the dimension side is broadcastable, which is why it silently stops working once the dimension grows past spark.sql.autoBroadcastJoinThreshold. The evidence is always the same two places: PartitionFilters on the FileScan node, and the number of files read plus Input Size / Records on the Stages page.

-- prunes
WHERE event_date >= '2024-01-01'
-- does not prune
WHERE year(event_date) = 2024

Senior depth: I read PartitionFilters before I read runtime. Pruning is binary and visible in the plan, so there is never a reason to speculate. The DPP-stops-working case is worth planning for: dimensions grow, broadcast eligibility disappears, and a query that pruned for a year starts full-scanning with no code change.

Common mistake: Writing a predicate against a derived expression of the partition column and reporting that 'partitioning does not work'.

Follow-up they will ask: How do you confirm dynamic partition pruning actually fired?

Lesson · Simulation

advanced

A query defines a CTE and references it three times; the source table is scanned three times. Why isn't a CTE a cache?

Spark SQL: the same engine, a different keyboard · tap to open the answer

Say this first: A CTE is a name for a subquery that Catalyst may inline at each reference — it carries no materialisation guarantee.

Remember these

  • A CTE is a logical name; Catalyst may inline it per reference
  • Shared subtrees ending in a shuffle can appear as ReusedExchange
  • A shuffle-free shared subtree is simply planned again
  • Use CACHE TABLE, persist(), or a temp table for real materialisation

Why: In Spark, a WITH clause introduces a logical name, and the optimizer is free to substitute the subquery at every reference site, which is usually what you want because it lets predicates and projections push into each use independently. The cost is that identical work can be planned more than once. Spark will sometimes avoid repeating it: when the shared subtree ends in a shuffle, the planner can reuse the exchange and you see ReusedExchange in the physical plan pointing back at the original Exchange. With no shuffle in the shared part — a plain filtered scan, for example — there is nothing to reuse and each reference becomes its own FileScan. If you need a real materialisation, say so: CACHE TABLE over the subquery, persist() on the DataFrame equivalent, or write a temporary table when the subquery is expensive and reused by several queries. Read the SQL tab's plan and count FileScan nodes against the same path; that number, not the CTE count, tells you what you are paying.

CACHE TABLE recent AS
SELECT * FROM orders WHERE order_ts > current_date() - INTERVAL 7 DAYS;
SELECT region, count(*) FROM recent GROUP BY region;

Senior depth: Inlining is usually the right default — a materialised CTE would block the pushdown that makes each reference cheap. The judgement call is whether the shared part is expensive and the per-reference pushdown is weak. Only then is paying for materialisation worth it, and then I prefer a written table over a cache so the cost is visible and bounded.

Common mistake: Treating a CTE as an optimisation fence that guarantees the subquery runs once.

Follow-up they will ask: How do you count how many times the same table is scanned in one query?

Lesson · Simulation

architect

A BI tool is about to fire four thousand small SQL queries an hour at your Spark cluster. What do you tell the team?

Spark SQL: the same engine, a different keyboard · tap to open the answer

Say this first: A batch Spark cluster is the wrong serving layer; route interactive SQL to a warehouse-style endpoint and pre-aggregate what the dashboard needs.

Remember these

  • Per-query planning and submission overhead dominates small queries
  • One shared driver makes planning, not CPU, the concurrency bottleneck
  • Batch Spark has no result cache; dashboards recompute every refresh
  • Pre-aggregated tables plus a warehouse endpoint is the serving pattern

Why: Every one of those queries pays full Catalyst planning, job and stage submission, and a shuffle whose partition count was tuned for batch, so a query touching a few megabytes still spends seconds in fixed overhead. Concurrency makes it worse rather than better: all sessions share one driver, so planning and the listener bus become the contention point long before executor CPU does, and one user's heavy query can stall everyone. Batch Spark also has no result cache, so the identical dashboard tile recomputes on every refresh. The architecture that works is a serving tier — a SQL warehouse endpoint with result caching and fast startup — fed by pre-aggregated Delta tables that the batch pipeline maintains. Push the heavy grouping into the nightly job so dashboard queries read narrow, already-aggregated rows with partition pruning and Z-ordering or liquid clustering on the filter columns. If a genuinely sub-second interactive requirement remains, that belongs in a serving database, not in Spark at all.

Senior depth: Draw the line at the grain the dashboard queries. If the BI tool never asks for row-level detail, no interactive query should ever touch the fact table. The architectural failure is letting the serving layer query the batch layer's physical model, and it will keep costing money regardless of how the compute is sized.

Common mistake: Scaling the batch cluster up to absorb dashboard concurrency, which does nothing about driver-side planning contention.

Follow-up they will ask: Which part of the dashboard workload would you still leave on the batch cluster?

Lesson · Simulation

senior

You chained 30 withColumn calls. The job is slow but the plan looks simple. What happened?

Narrow vs wide transformations · tap to open the answer

Say this first: Each withColumn can add projection overhead; UDFs in those columns serialize every row.

Why: Prefer select with expressions. If columns are Python UDFs, you lost Tungsten/Photon. Spark UI: time in Python vs CPU.

Common mistake: More executors for a UDF pipeline.

Follow-up they will ask: How do you rewrite 30 withColumns cleanly?

Lesson · Simulation

advanced

If Catalyst pushes predicates anyway, does it matter whether you filter before or after a join?

Narrow vs wide transformations · tap to open the answer

Say this first: For inner joins it usually does not; for outer joins, aggregates, and windows the optimizer cannot push through, so placement changes both cost and results.

Remember these

  • Inner-join predicates referencing one side are pushed automatically
  • A filter on the null-producing side of an outer join cannot be pushed
  • Aggregates pass predicates on grouping keys only — WHERE versus HAVING
  • Windows and Python UDFs are hard barriers to pushdown

Why: PushDownPredicate moves a filter below an inner join whenever the predicate references only one side, so the two spellings converge on the same physical plan. Outer joins break that: a filter on the null-producing side cannot be pushed without changing semantics, because the join is what manufactures those nulls — which is also why a where clause on the right side of a left outer join silently turns it into an inner join. Aggregates only let predicates through when they reference grouping keys, never aggregate outputs, which is the difference between WHERE and HAVING. Window functions are a hard barrier, since a filter above a window would change the frame the ranking was computed over. Python UDFs are opaque, so any predicate wrapping one stays where you put it. The practical rule is to filter as early as you can express it correctly and then verify with explain('formatted') that the predicate ended up in PushedFilters rather than in a Filter node above the Exchange.

# changes an outer join into an inner one
left.join(right, 'id', 'left').filter(right.status == 'OK')
# keeps outer semantics
left.join(right.filter("status = 'OK'"), 'id', 'left')

Senior depth: The dangerous case is not the slow one, it is the one where pushdown is blocked for correctness and the author did not notice the semantics changed. I read every outer join's downstream filters in review, because that is the single most common silently-wrong pattern in DataFrame code.

Common mistake: Trusting the optimizer uniformly and shipping a left outer join that a later where clause quietly made inner.

Follow-up they will ask: How would you keep outer-join semantics while still filtering the right side early?

Lesson · Simulation

senior

An explode on a nested events array takes down one executor every run. What is the mechanism and how do you bound it?

Narrow vs wide transformations · tap to open the answer

Say this first: explode multiplies rows inside the task that holds the parent row, so a single fat array produces a huge partition that no later repartition can rescue.

Remember these

  • explode plans as Generate and is narrow — no redistribution
  • One oversized array is one oversized task, whatever the partition count
  • repartition after the explode is too late to help
  • Fix upstream: more input parallelism, or isolate the fat records

Why: Generate — the plan node behind explode — is narrow: output rows are produced in the same task as their input row, with no opportunity to redistribute. So if one record carries an array of two million elements, one task must materialise all of it, and that task's memory pressure is what kills the executor, regardless of how many partitions the stage has overall. Adding repartition after the explode does not help, because the explosion has already happened by the time the shuffle sees the rows. The levers are all upstream. Measure size(col('events')) and look at the distribution before anything else, then increase input parallelism so each task holds fewer parent rows — a smaller spark.sql.files.maxPartitionBytes or an explicit repartition before the explode. Filter or cap pathological arrays if the business allows it, and if a few records are genuinely enormous, split them out and process that tail separately. The UI signature is specific: one task in a narrow stage with Peak Execution Memory far above the median, no shuffle involved.

df.select(size('events').alias('n')).summary('min', '75%', 'max').show()
df.repartition(4000).select(explode('events').alias('e'))

Senior depth: Any row-multiplying operator needs its fan-out measured before it ships, the same way you would size a join. The failure is always in the tail of the array-length distribution, so the max matters far more than the average, and the memory ceiling is per-task — which is why this is one of the few problems a bigger cluster genuinely cannot solve.

Common mistake: Adding repartition immediately after the explode and reporting that Spark 'ignored' it.

Follow-up they will ask: How would you process the handful of records with million-element arrays?

Lesson · Simulation

senior

display() in Databricks feels harmless. When is it collect in disguise?

Actions that trigger jobs · tap to open the answer

Say this first: When the notebook pulls a large unaggregated result to render.

Why: UI limits help, but a wide table with huge strings still hammers the driver. Prefer aggregations or LIMIT. Job clusters should not display fact tables.

Common mistake: Leaving display(df) in a scheduled notebook.

Follow-up they will ask: How do you QA a 2 TB write without pulling it?

Lesson · Simulation

advanced

take(1000) on a filtered table submitted eleven jobs. Explain the mechanism.

Actions that trigger jobs · tap to open the answer

Say this first: Limit scale-up: Spark scans a small partition set, finds too few matching rows, multiplies the partition count by spark.sql.limit.scaleUpFactor, and runs again.

Remember these

  • CollectLimit grows the scanned partition set between attempts
  • Each attempt is a separate job under one SQL query ID
  • Rare matching rows make the total work approach a full scan
  • Prunable predicates fix it; a bigger scale-up factor does not

Why: CollectLimit is deliberately incremental. It starts with one partition, and if the rows collected so far are fewer than the requested limit, it retries with the partition count multiplied by spark.sql.limit.scaleUpFactor, repeating until it has enough rows or has exhausted the table. Each of those attempts is a separate job under the same SQL query, which is exactly the pattern you saw: eleven jobs with growing task counts and growing Input Size / Records. When the filter is unselective this terminates on the first or second attempt and the action feels free. When the matching rows are rare or clustered at the end of the file listing, every attempt fails to fill the limit and the total work approaches a full scan plus the overhead of re-planning and re-launching repeatedly. The fix is to make the predicate prunable rather than to tune the factor — a partition filter, Delta data skipping, or clustering on the filter column turns the first attempt into a hit. Raising the scale-up factor only makes each attempt bigger; it does not reduce the number of rows that must be found.

spark.conf.get('spark.sql.limit.scaleUpFactor')
df.filter("event_date = '2024-05-01' AND rare_flag").take(1000)

Senior depth: Scale-up is the engine making a reasonable bet that a limit is cheap. The lesson for pipeline design is that take and show are not constant-cost probes — their cost is a function of predicate selectivity, which is why a sampling helper that works on bronze can quietly become a full scan on a filtered gold table.

Common mistake: Reading the repeated jobs as retries from a failure rather than as limit scale-up attempts.

Follow-up they will ask: Why does adding a partition filter change the number of attempts?

Lesson · Simulation

senior

A team wants to push rows to an external API with foreachPartition. What goes wrong, and what would you do instead?

Actions that trigger jobs · tap to open the answer

Say this first: Task retries and speculation re-run partitions, so the API receives duplicates; the write needs to be idempotent or go through a staged table.

Remember these

  • Task retries and speculation can execute a partition more than once
  • Spark has no transaction boundary around external side effects
  • Idempotency keys or a staged table are the only safe patterns
  • Hundreds of tasks mean hundreds of concurrent connections

Why: foreachPartition runs your function on executors with one call per partition, which correctly amortises connection setup compared with foreach's per-row cost. The problem is the execution model: a task can fail and be retried up to spark.task.maxFailures, and with spark.speculation enabled a slow task can be run twice concurrently and the loser's work is discarded by Spark but not by your API. Spark has no transaction boundary around external side effects, so there is no at-most-once guarantee and partial partitions are visible to the outside world. The design that survives this is idempotency: send a deterministic idempotency key per row so the server can deduplicate, or write to a staging Delta table and have a separate, single-threaded process drain it with committed offsets. Add bounded concurrency and retries with backoff inside the function too, because a few hundred tasks opening connections at once is a load spike the API owner did not agree to. In streaming the equivalent is foreachBatch, which at least gives you a batch id you can use as the idempotency key.

def send(rows):
    with client() as api:
        for r in rows:
            api.put(key=f'{r.id}:{r.version}', body=r.asDict())

df.foreachPartition(send)

Senior depth: I treat any external write from executors as at-least-once by definition and design the receiver accordingly. The alternative that actually scales is to stop writing from executors: land the rows in a table, then let one owner with checkpointed offsets push them out, so retry semantics live in one place you can reason about.

Common mistake: Assuming a successful job means every row was delivered exactly once.

Follow-up they will ask: What would you use as the idempotency key for a streaming source?

Lesson · Simulation

senior

Two actions on the same DataFrame returned two different results in one run. How?

Actions that trigger jobs · tap to open the answer

Say this first: Each action re-executes the plan, so a changing source or a non-deterministic expression yields a different answer the second time.

Remember these

  • Every action re-executes the plan from the source
  • Concurrent appends mean two actions read two table versions
  • current_timestamp() and unseeded rand() differ per execution
  • Pin the input with a Delta version; cache is not a snapshot guarantee

Why: Without a cache or a checkpoint, a DataFrame is a recipe, and every action cooks it again from the source. If a concurrent job appended to the input table between your count() and your write(), the two executions read different table versions and both numbers are correct for their moment. The same thing happens with no concurrency at all when the plan contains current_timestamp(), rand() without a seed, or a filter relative to the wall clock, because those are evaluated per execution. The honest fix is to pin the input rather than to cache the output: read a specific Delta version or timestamp so every action in the run sees one snapshot. Caching makes the symptom less likely but is not a guarantee, since a cached partition can be evicted and recomputed from the source. Logging the Delta version the run consumed alongside the row count is what makes the result reproducible after the fact.

v = spark.sql('DESCRIBE HISTORY bronze.orders').first()['version']
src = spark.read.option('versionAsOf', v).table('bronze.orders')

Senior depth: Reproducibility comes from pinning inputs, not from caching intermediates. Any pipeline whose output I might have to defend reads an explicit table version and records it in the run metadata, so 'why did last Tuesday's number change' has an answer that does not depend on what was in memory.

Common mistake: Adding cache() to make two actions agree, and treating that as a correctness guarantee.

Follow-up they will ask: Why is cache() not a snapshot?

Lesson · Simulation

senior

When is laziness a production footgun?

Why Spark is lazy · tap to open the answer

Say this first: When invalid tables, bad schemas, or huge plans only fail at the action — in a job cluster at 2am.

Why: Validate schema and table existence early. Unit-test explain() on representative data. Don't leave 15 debug actions that each rerun a shuffle.

Common mistake: Adding count() everywhere as 'validation'.

Follow-up they will ask: What is a cheap eager check that isn't a full shuffle?

Lesson · Simulation

advanced

When is breaking lineage with checkpoint() the right answer instead of cache()?

Why Spark is lazy · tap to open the answer

Say this first: When the plan itself is the problem — an iterative loop whose lineage keeps growing — because checkpoint truncates the plan while cache only stores results under it.

Remember these

  • cache() keeps the lineage; checkpoint() replaces it with a file read
  • Growing plans in iterative loops are a planning problem, not a data problem
  • localCheckpoint() truncates lineage but is not fault tolerant
  • Writing an intermediate Delta table is often the better checkpoint

Why: cache() stores the computed partitions but leaves the full lineage attached, so the plan the driver carries keeps growing and any evicted block is recomputed through that entire history. checkpoint() writes the dataset to the directory set by sparkContext.setCheckpointDir and replaces the plan with a plain read of those files, so the lineage is genuinely cut and recovery reads from reliable storage. That matters for loops: a feature-engineering or graph-iteration job that appends operators every pass eventually spends more time in Catalyst than in execution, and no amount of caching fixes a plan that is thousands of nodes deep. localCheckpoint() truncates the lineage without the durable write, which is faster and loses the data if an executor is lost — fine inside a single job, wrong as a recovery point. Both are actions in their eager form, so they cost a full execution at the point you call them. In a lake pipeline, writing an intermediate Delta table is often the better version of the same idea, because it is inspectable, reusable by the next run, and visible in table history.

spark.sparkContext.setCheckpointDir('/tmp/ckpt')
for i in range(50):
    df = df.withColumn(f'f{i}', expr('amount * 1.01'))
    if i % 10 == 0:
        df = df.checkpoint()

Senior depth: The question I ask is what exactly is expensive: the bytes or the tree. Cache addresses bytes, checkpoint addresses the tree. Measuring the gap between query submission and the first task launch tells you which one you have, and that measurement takes ten seconds while guessing wrong costs a day.

Common mistake: Reaching for cache() against a driver that is slow at planning, when the problem is plan depth rather than recomputation.

Follow-up they will ask: How would you tell planning time from execution time in the UI?

Lesson · Simulation

senior

You want the pipeline to fail fast on a bad schema at 02:00 instead of ninety minutes in. What cheap eager checks do you add?

Why Spark is lazy · tap to open the answer

Say this first: Metadata-only assertions up front: resolve every source table, compare schemas, probe one row per input, and assert the plan's pushdown before the heavy work starts.

Remember these

  • Resolving a table's schema is a catalog call, not a job
  • limit(1).count() per source costs one task and catches empty inputs
  • Assert PartitionFilters in the plan to catch pruning regressions
  • Delta constraints enforce data rules on every write, not just this job

Why: Lazy evaluation means a missing table or a renamed column only surfaces when an action reaches that part of the plan, so a validation block at the top of the job converts a late failure into an immediate one. Resolving spark.table(name).schema for every input costs a catalog round trip and no stage, and comparing it against an expected StructType catches renames, type drift, and added columns before any data moves. A bounded existence probe — limit(1).count() or isEmpty() on each source — costs one task per input and catches empty or unreadable partitions. Calling explain('formatted') and asserting that PartitionFilters is non-empty on the big fact scan catches the day someone wraps the partition column in a cast and turns a pruned read into a full scan. For data-level guarantees, let the storage layer carry them: Delta NOT NULL and CHECK constraints are validated on every write by every writer, which is stronger than any assertion in one job. What you should not add is a count() per transformation, because that is a full re-execution dressed up as a check.

expected = spark.table('contracts.orders_v3').schema
actual = spark.table('bronze.orders').schema
assert actual == expected, f'schema drift: {actual}'

Senior depth: Separate the two failure classes. Contract failures — missing table, wrong schema, lost pruning — are cheap to detect and belong at the top of the job. Data failures belong in table constraints and expectations so they are enforced for every writer. Mixing them produces jobs full of count() calls that cost real money and still miss the thing that broke.

Common mistake: Calling it validation while sprinkling count() between transformations, which re-runs the plan every time.

Follow-up they will ask: Which of these checks belongs in the pipeline and which belongs in the table definition?

Lesson · Simulation

architect

A team wants to pass a DataFrame from one Workflow task to the next instead of writing a table. Explain why that cannot work.

Why Spark is lazy · tap to open the answer

Say this first: A DataFrame is a handle into one live SparkSession; a separate task is a different session and usually a different JVM, so there is nothing to hand over.

Remember these

  • A DataFrame references a live session and plan; it cannot be serialised
  • Each task gets its own SparkSession and its own session catalog
  • The durable handoff is a table name plus a version, not an object
  • Snapshot identifiers at task boundaries make partial reruns deterministic

Why: The object holds a reference to a session, a plan tree, and driver-side state such as cached relations — none of which is serialisable or meaningful outside the process that built it. Even where two tasks share a cluster, each gets its own SparkSession with its own session catalog, so temp views and caches from the first are invisible to the second. The only durable handoff is something both sessions can resolve independently: a table name, a path, or a table name plus a version. Delta makes that handoff precise, because the producing task can emit the committed version and the consumer can read exactly that snapshot, which also makes a partial rerun deterministic. Treat each task boundary as a contract with a schema and a snapshot identifier, and the pipeline becomes restartable at any step instead of all-or-nothing. If the intermediate genuinely must not be durable, the honest answer is that it belongs inside one task, not split across two.

# task 1
dbutils.jobs.taskValues.set('version', version)
# task 2
v = dbutils.jobs.taskValues.get('etl', 'version')
spark.read.option('versionAsOf', v).table('silver.orders')

Senior depth: Task boundaries are where restartability is won or lost. Durable intermediates with explicit versions cost storage and buy you the ability to rerun step four at 03:00 without rerunning steps one through three. That trade is almost always worth it, and the teams that resist it are usually optimising a cost they have never measured.

Common mistake: Trying to share state across tasks with a global temp view and assuming it survives because the cluster is the same.

Follow-up they will ask: What should the task boundary contract contain besides the table name?

Lesson · Simulation

senior

How do you size partitions for a shuffle?

Partitions: the unit of parallelism · tap to open the answer

Say this first: Aim for ~100–256 MB per shuffle partition, then look at the histogram.

Why: Too few: fat tasks, spill, OOM. Too many: scheduler overhead. AQE coalesce helps small data. Skew means one partition will still be fat — salting, not more partitions, fixes that.

Senior depth: You want enough partitions for cores × duration, sized by data bytes, not by machine count.

Common mistake: Setting shuffle partitions to executor_count.

Follow-up they will ask: Why is executor_count the wrong formula?

Lesson · Simulation

advanced

A stage runs 200 tasks on 160 cores. Is 200 a good number? Answer without guessing.

Partitions: the unit of parallelism · tap to open the answer

Say this first: Judge it by bytes per task and by how many scheduling waves it creates, not by the number itself.

Remember these

  • Target roughly 100-200 MB of shuffle read per task, then stop tuning
  • Partition count that is not a multiple of total cores leaves a ragged wave
  • Summary Metrics max vs median exposes skew that recounting cannot fix
  • Task durations in the hundreds of milliseconds mean scheduler overhead dominates

Why: Two constraints decide the count. First, bytes: open the Stage page, read Summary Metrics for Shuffle Read, and aim for the median task to land in the low hundreds of MB so tasks run for seconds rather than milliseconds or minutes. Second, waves: 200 tasks on 160 cores is one full wave plus a 40-task straggler wave, so 160 cores are busy for the first wave and 120 idle for the second. Rounding to a multiple of the core count removes that ragged tail. Then check the max-versus-median column in Summary Metrics, because an even count with a 50x max/median ratio is a skew problem that no partition count will fix.

spark.conf.set("spark.sql.shuffle.partitions", 320)  # 2 waves on 160 cores

Senior depth: With AQE on, treat the configured count as an upper bound you deliberately overshoot. Set spark.sql.shuffle.partitions high, enable spark.sql.adaptive.coalescePartitions.enabled, and let spark.sql.adaptive.advisoryPartitionSizeInBytes merge the small ones at runtime. You tune a target size once instead of a count per job.

Common mistake: Quoting '200 is the default so 200 is fine' instead of reading bytes per task off the Stage page.

Follow-up they will ask: How does AQE change this calculation?

Lesson · Simulation

senior

AQE is enabled and your post-shuffle stage still shows exactly 200 tasks. What does that tell you?

Partitions: the unit of parallelism · tap to open the answer

Say this first: Either coalescing is off, or the partitions are already at or above the advisory size so there is nothing to merge.

Remember these

  • Coalescing needs spark.sql.adaptive.coalescePartitions.enabled too
  • Look for AQEShuffleRead above the Exchange in the SQL tab to confirm it ran
  • AQE merges partitions downward; it does not create extra tasks
  • Scan stages are never coalesced by AQE

Why: AQE only reshapes partitions after a shuffle has written its map output statistics, and coalescing requires spark.sql.adaptive.coalescePartitions.enabled on top of spark.sql.adaptive.enabled. When it runs, it merges adjacent small partitions until each reaches spark.sql.adaptive.advisoryPartitionSizeInBytes, and the Spark UI SQL tab shows a CustomShuffleReader or AQEShuffleRead node above the Exchange. Seeing a clean 200 means one of three things: AQE is off for this query, the stage is not post-shuffle at all (a scan is never coalesced), or every partition is already larger than the advisory size so merging would do nothing. Note that AQE coalesces downward only — it never splits a partition to create more tasks, except in the specific skew-join path.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128MB")

Senior depth: No. Coalescing cannot split. To get smaller post-shuffle partitions you raise the configured shuffle partition count so the Exchange produces more of them, and let AQE merge back whatever turns out too small. Overshoot plus coalesce is the pattern; undershoot plus AQE is a dead end.

Common mistake: Assuming AQE is active because the cluster default says so, without checking the plan for the AQE shuffle read node.

Follow-up they will ask: Your partitions are 1 GB each and you want them smaller. Does AQE help?

Lesson · Simulation

architect

The platform team wants one cluster-wide spark.sql.shuffle.partitions for every job. What do you push back with?

Partitions: the unit of parallelism · tap to open the answer

Say this first: A fixed count is wrong for every job except the one it was tuned on, because the invariant people actually want is bytes per partition.

Remember these

  • Bytes per partition is portable across jobs; a partition count is not
  • advisoryPartitionSizeInBytes is the right platform-level default
  • Set the static count high and let AQE coalesce downward
  • Per-job overrides belong with the job code, not in cluster policy

Why: A count is a function of data volume, and volume differs by orders of magnitude across a platform. The same 200 that gives a 2 TB job sane 1 GB partitions gives a 2 GB job eight-megabyte partitions and thousands of millisecond tasks whose scheduler delay exceeds their compute. The portable default is a target size: turn on spark.sql.adaptive.enabled with coalescePartitions, set spark.sql.adaptive.advisoryPartitionSizeInBytes once as the platform standard, and set the static count generously high so AQE has material to merge. Individual jobs then override only when they have a measured reason, and that override lives in the job config next to the code, not in a cluster policy nobody can trace. The one thing a single global number does buy you is predictability for capacity planning, which is worth saying out loud before you argue against it.

Senior depth: Mine the Spark event logs. Every stage records task count, shuffle read bytes, and task durations, so you can compute median bytes per task and max/median duration ratio per stage and rank the whole platform by it. That turns partition tuning from a per-ticket argument into a dashboard with a worst-offenders list.

Common mistake: Arguing from folklore numbers instead of naming the invariant the config is standing in for.

Follow-up they will ask: How would you detect, across hundreds of jobs, which ones are badly partitioned?

Lesson · Simulation

senior

You have a job that joins on customer_id and then windows on customer_id. When is repartition('customer_id') worth paying for?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: When one shuffle on that key gets reused by more than one downstream operator.

Why: repartition('customer_id') hash-partitions rows so matching keys are co-located, and a following join or window on the same key can then skip its own Exchange. If only one operator needs the key, it would have shuffled once anyway — you have just paid for two shuffles instead of one.

Senior depth: explain('formatted') and count Exchange nodes before and after the change. If the count did not drop, the repartition bought you nothing.

Common mistake: Adding repartition(col) before every join out of habit and doubling the shuffle count.

Follow-up they will ask: How do you verify the second shuffle actually disappeared?

Lesson · Simulation

architect

spark.sql.shuffle.partitions is 200 and the code also calls repartition(1000). Which one wins, and where?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: They have different scopes: the config sizes the engine's own Exchanges, repartition(1000) sizes that one explicit shuffle.

Why: Every groupBy or sort-merge join Exchange targets spark.sql.shuffle.partitions unless AQE coalesces it at runtime; repartition(1000) is its own Exchange with its own target, and a later groupBy re-shuffles back to the configured number. df.write.partitionBy is a third, unrelated thing — it controls the on-disk directory layout, not in-memory partition count.

Senior depth: repartition on the same columns you pass to write.partitionBy, so each task owns one output directory and writes one decent-sized file instead of every task dropping a fragment into every directory.

Common mistake: Conflating write.partitionBy with repartition and expecting one to control the other.

Follow-up they will ask: How do you get a sane directory layout and reasonable file sizes at the same time?

Lesson · Simulation

advanced

In explain output you see Exchange RoundRobinPartitioning(200) in one plan and Exchange hashpartitioning(customer_id, 200) in another. What changes for the stage that follows?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Hash partitioning satisfies a downstream operator's distribution requirement, so the planner can drop its Exchange; round-robin cannot.

Remember these

  • EnsureRequirements removes an Exchange when the child partitioning already satisfies it
  • Hash partitioning on the key co-locates; round-robin only balances
  • Partition counts must match for the downstream Exchange to be dropped
  • Count Exchange nodes before and after to prove the optimization landed

Why: Every operator that needs co-location declares a required child distribution, and the EnsureRequirements rule inserts an Exchange only when the child's existing partitioning does not satisfy it. hashpartitioning(customer_id, 200) satisfies a following groupBy, SortMergeJoin, or Window on customer_id with the same partition count, so no second Exchange appears and you have paid for one shuffle instead of two. RoundRobinPartitioning satisfies nothing except 'some distribution', so the next wide operator shuffles again. The practical way to check is to count Exchange nodes in explain('formatted') before and after the change; if the count did not drop, the keyed repartition bought you nothing and you are now paying an extra shuffle for even-sized partitions you could have gotten for free. Round-robin also used to be a correctness hazard on retry because the row-to-partition assignment is not a function of the data, which is why Spark sorts within round-robin partitions before writing shuffle output.

df.repartition(200, "customer_id").groupBy("customer_id").sum("amount").explain("formatted")

Senior depth: The counts no longer match, so the groupBy's required distribution is not satisfied and the planner reinserts an Exchange at 400. You have paid for two shuffles and saved none. Either align the repartition count with the shuffle partition count or let the engine plan the single Exchange itself.

Common mistake: Assuming any repartition before a groupBy saves a shuffle, regardless of columns or count.

Follow-up they will ask: What happens if you repartition to 200 but spark.sql.shuffle.partitions is 400?

Lesson · Simulation

advanced

Your output volume swings ten-fold between a weekday and a sale day. repartition(64) gives 2 MB files on Tuesday and 2 GB files on Black Friday. How do you get stable file sizes?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: Stop pinning a partition count and bound the write by rows or bytes per file instead.

Remember these

  • A fixed partition count fixes file count, not file size
  • maxRecordsPerFile rolls a new file within one write task
  • AQE coalescing handles the small-volume direction
  • Compaction is the backstop, not the primary control

Why: A fixed repartition count fixes the number of output files, which means file size is whatever the day's volume divided by that count happens to be. The writer-side control is maxRecordsPerFile, which makes a single write task roll to a new file once it has emitted that many rows, so a fat partition produces several right-sized files rather than one huge one. For the small-day direction, enable AQE coalescing so tiny post-shuffle partitions merge before the write instead of each becoming a fragment. On Delta, optimized writes and auto-compaction do this adaptively at the table level, and a scheduled OPTIMIZE fixes whatever still slips through. Compute the repartition count from an estimated byte volume if you must pick one, but treat the per-file cap as the thing that actually holds the contract.

(df.write
   .option("maxRecordsPerFile", 2_000_000)
   .mode("overwrite")
   .parquet("/mnt/curated/orders"))

Senior depth: Write one sample batch, divide output bytes by output rows to get an average compressed bytes-per-row, then divide your target file size by that. Re-check it after any schema change that adds wide columns, because bytes-per-row is the thing that drifts.

Common mistake: Hand-tuning the repartition count after every incident instead of bounding file size directly.

Follow-up they will ask: How would you pick the maxRecordsPerFile value for a table you have never written before?

Lesson · Simulation

senior

You filter a 2 TB table down to 3 GB and then call coalesce(50) before writing. The write looks fine but the job got much slower. What happened?

Repartition vs coalesce vs partitionBy · tap to open the answer

Say this first: coalesce has no shuffle boundary, so the 50 propagated back up into the scan and 50 tasks now read the full 2 TB.

Remember these

  • coalesce is narrow, so its partition count propagates up the whole narrow chain
  • The throttled stage shows huge Input Size with no shuffle metrics to blame
  • repartition(n) adds the Exchange that stops the propagation
  • The propagation stops only at the nearest upstream shuffle boundary

Why: coalesce creates a narrow dependency: each output partition is computed by one task that reads several parent partitions. Because there is no Exchange, there is no stage boundary either, so the filter and the scan feeding it are fused into the same stage as the write — and that stage has 50 tasks. Those 50 tasks are now responsible for scanning all 2 TB, not the 3 GB that survives the filter. The Spark UI shows one long stage with 50 tasks and a huge Input Size / Records, and nothing in the shuffle metrics to explain the time, which is exactly why this one is hard to spot. The fix is repartition(50), which inserts an Exchange so the scan keeps its original parallelism and only the post-shuffle write runs at 50.

big.filter("event_date = '2026-10-23'").coalesce(50).write.parquet(path)   # scan runs at 50
big.filter("event_date = '2026-10-23'").repartition(50).write.parquet(path) # scan keeps parallelism

Senior depth: When the stage it collapses into is already cheap — typically when the filter is a partition-pruning predicate so the scan reads little, or when the upstream work is already behind a shuffle boundary that coalesce cannot cross. Check where the nearest Exchange is before you decide; that boundary is the limit of the damage.

Common mistake: Reading coalesce as 'applies at the write' because that is where the call sits in the code.

Follow-up they will ask: When is coalesce after a filter actually the right call?

Lesson · Simulation

senior

How do you remove a shuffle from a join?

What a shuffle actually does · tap to open the answer

Say this first: Broadcast the small side, or co-partition/bucket both sides on the join key.

Why: BroadcastHashJoin keeps the fact table in place. SortMergeJoin shuffles both unless already partitioned the same way. Prove with Exchange absence in explain().

Senior depth: When both sides are large and unsorted. Then make it balanced — not broadcast a 30 GB dimension.

Common mistake: spark.sql.shuffle.partitions as the join strategy.

Follow-up they will ask: When is a shuffle the right answer?

Lesson · Simulation

advanced

A task fails with FetchFailedException. Whose failure is that, and what does Spark do next?

What a shuffle actually does · tap to open the answer

Say this first: A reducer could not fetch map output, so Spark treats the map stage as lost and reruns the missing map tasks before retrying the reduce stage.

Remember these

  • The exception surfaces on the reducer but the cause sits on the map side
  • Spark reruns the missing map tasks, then retries the reduce stage
  • spark.stage.maxConsecutiveAttempts bounds the retry loop
  • External shuffle service keeps map output alive after executor loss

Why: FetchFailedException is raised on the reduce side but the cause is almost always on the map side: the executor that wrote those shuffle blocks is gone, its local disk filled or failed, the node was decommissioned or reclaimed as a spot instance, or the fetch timed out under load. The DAGScheduler responds by marking the map stage's output for that executor as unavailable, resubmitting only the missing map tasks, and then rerunning the reduce stage — which is why you see a stage 'retry 1' with far fewer tasks than the original. This repeats up to spark.stage.maxConsecutiveAttempts before the job fails outright. Running an external shuffle service decouples shuffle files from executor lifetime so losing an executor no longer loses its map output, and on Kubernetes where there is no shuffle service the equivalent is dynamic allocation shuffle tracking, which keeps executors alive while their shuffle data is still referenced.

Senior depth: Keep the driver and a core group of executors on on-demand capacity and let only the expansion capacity be preemptible, enable the shuffle service so a reclaimed node does not drag its map output with it, and shorten the shuffle stages — the less map output alive at any moment, the less a reclaim costs. Raising retry counts only makes the failure take longer.

Common mistake: Raising network timeouts first, when the executor that held the blocks is simply gone.

Follow-up they will ask: You are on spot instances and this happens nightly. What is your first structural change?

Lesson · Simulation

advanced

One task shows Spill (Memory) of 40 GB and Spill (Disk) of 6 GB. Which number should worry you, and what do they actually measure?

What a shuffle actually does · tap to open the answer

Say this first: They are the same data measured twice — deserialized in memory versus serialized on disk — and the disk number is the IO you actually pay for.

Remember these

  • Spill Memory is deserialized size; Spill Disk is serialized bytes written
  • Their ratio is the serialization factor, not an error
  • Disk spill is the number that costs wall clock through write, read, and merge
  • Spill on one task only is a skew signal, not a memory-config signal

Why: When a task's execution memory fills, the sorter serializes its in-memory records and writes them to local disk. Spill (Memory) is the size of those records as live JVM objects before the write; Spill (Disk) is the compressed, serialized bytes that landed. The ratio between them is your serialization and compression factor, usually somewhere around 5-10x for typical row shapes, so a wide gap is normal and not itself a bug. The number that costs wall clock is the disk one, because it is real write-then-read IO plus the merge pass afterwards. The real signal, though, is that spill is happening at all on one task: it means that partition does not fit in the execution memory available per task, so the fix is more partitions, less data per key, or fewer concurrent tasks per executor — not a bigger spill buffer.

Senior depth: Under unified memory management, execution can evict cached storage blocks down to the storage reservation, but storage cannot evict execution. So a job that both caches aggressively and shuffles heavily will quietly lose its cache first and start recomputing — which looks like a mysterious slowdown rather than a memory error. Check the Storage tab fraction-cached alongside the spill metrics before changing any memory fraction.

Common mistake: Reporting the memory spill number as 'we need 40 GB more executor memory'.

Follow-up they will ask: Execution memory and storage memory compete. Which one gives way first?

Lesson · Simulation

senior

How does the external shuffle service change what dynamic allocation is allowed to do?

What a shuffle actually does · tap to open the answer

Say this first: It serves shuffle blocks from a process outside the executor, so idle executors can be released without destroying their map output.

Remember these

  • Shuffle files outlive the executor only when a separate daemon serves them
  • Without it, scaling down an executor invalidates its map output
  • spark.shuffle.service.enabled is the YARN-era companion to dynamic allocation
  • shuffleTracking is the K8s alternative and limits how far you can scale down

Why: Without it, an executor is the only thing that can serve the shuffle files it wrote, so releasing an idle executor means any reducer that later needs those blocks hits FetchFailedException and triggers a map-stage rerun. The external shuffle service runs one long-lived daemon per node, holding the shuffle directories, so scaling down is safe and spark.dynamicAllocation.enabled can actually shrink the cluster between stages. That is why spark.shuffle.service.enabled and dynamic allocation are historically mentioned in the same breath on YARN. On Kubernetes there is typically no node-level daemon, so the equivalent is spark.dynamicAllocation.shuffleTracking.enabled, which simply refuses to release an executor while its shuffle output is still referenced — safer, but it means heavy-shuffle jobs scale down much less than you would hope.

Senior depth: Shuffle tracking is holding them. Those executors still own shuffle blocks that a later stage may fetch, so the allocation manager will not release them. The real fix is structural — fewer or smaller shuffles, or an intermediate table written between phases so the shuffle data stops being live — rather than forcing the timeout down and reintroducing fetch failures.

Common mistake: Enabling dynamic allocation on a shuffle-heavy job with no shuffle service and then blaming the fetch failures on the network.

Follow-up they will ask: Your K8s job never scales below 40 executors despite long idle gaps. Why?

Lesson · Simulation

architect

One job reads a 3 TB fact table and groups it three times at three different grains. How do you cut the shuffle cost?

What a shuffle actually does · tap to open the answer

Say this first: Shuffle once to the finest grain you need, materialize that, and roll the coarser grains up from the small intermediate.

Remember these

  • Nested grains let one fine aggregate serve every coarser roll-up
  • sum, count, min, max compose; average composes only as sum plus count
  • Exact distinct does not roll up; approximate sketches do
  • A materialized fine-grained aggregate becomes a reusable table, not just a tuning hack

Why: Three independent groupBys mean three Exchanges that each move most of the 3 TB. If the three grains are nested — say customer-day, customer-month, and region-month — then the customer-day aggregate is a sufficient statistic for the other two, and the two roll-ups shuffle gigabytes instead of terabytes. This works cleanly for additive measures: sum, count, min, and max all roll up correctly. It breaks for averages unless you carry sum and count separately and divide at the end, and it breaks outright for exact distinct counts, which are not composable — those need either a separate pass or an approximate sketch like approx_count_distinct that does compose. Materializing the fine-grained aggregate as a real table also turns a tuning fix into a reusable asset, since the next team that needs a new grain rolls up from it rather than re-reading the fact.

daily = fact.groupBy("customer_id", "event_date").agg(F.sum("amount").alias("amt"), F.count("*").alias("n"))
daily.write.mode("overwrite").saveAsTable("curated.customer_daily")
monthly = daily.groupBy("customer_id", F.trunc("event_date", "MM")).agg(F.sum("amt"), F.sum("n"))

Senior depth: Then pick the single finest common grain that all three can be derived from, even if no consumer asks for it directly, and accept that it is larger than any individual output. If even that is impractical, cache or checkpoint the shared upstream so the three branches at least stop re-reading the fact table, and let each pay its own Exchange from there.

Common mistake: Rolling up an average of averages and shipping a number that is quietly wrong.

Follow-up they will ask: What do you do when the three grains are not nested at all?

Lesson · Simulation

senior

How do you debug a join that 'exploded' row counts?

Join strategies in Spark · tap to open the answer

Say this first: It's a many-to-many — duplicates on the key, not Spark magic.

Why: Count distinct keys vs rows on both sides before the join. Look for duplicate dimension keys. Spark UI won't show 'duplicates'; data profiling will.

Senior depth: SELECT key, COUNT(*) FROM dim GROUP BY 1 HAVING COUNT(*) > 1 — then the same on the fact.

Common mistake: Raising shuffle partitions to fix a row explosion.

Follow-up they will ask: What's the SQL test you'd run in the interview whiteboard?

Lesson · Simulation

advanced

Both sides of your join have about 30% NULL join keys and one task runs forever. Explain the mechanism and the fix.

Join strategies in Spark · tap to open the answer

Say this first: NULLs all hash to the same shuffle partition, so a third of both tables lands on one reducer doing work that can never match.

Remember these

  • All NULL keys hash to one partition and converge on one reducer
  • NULL never matches NULL in an equi-join, so that work produces nothing
  • Inner joins: filter nulls on both sides before the join
  • Outer joins: split, join the non-null rows, union the null-keyed rows back

Why: Partition assignment is the hash of the key, and every NULL hashes identically, so all NULL-keyed rows from both sides converge on one reducer. Under SQL equality semantics NULL never equals NULL, so none of those rows can produce an inner-join match — the reducer sorts and merges a third of your data to emit nothing. The Spark UI shows one task with enormous Shuffle Read and heavy Spill (Disk) while the rest of the stage finishes quickly. For an inner join the fix is a filter on both sides before the join, which also lets the optimizer push a null-safety predicate into the scan. For an outer join you cannot drop them, so handle the NULL population separately: join the non-null rows, then union back the null-keyed rows with the outer columns filled as nulls. Sentinel values like 'UNKNOWN' or -1 behave exactly the same way and are easy to miss because they do not look like nulls.

matched = left.where("customer_id IS NOT NULL").join(right.where("customer_id IS NOT NULL"), "customer_id", "inner")
unmatched = left.where("customer_id IS NULL")
result = matched.unionByName(unmatched, allowMissingColumns=True)

Senior depth: Group the join key and look at the top of the frequency distribution before you join. A single key holding double-digit percent of the rows is the tell, whether it is NULL, an empty string, a zero, or a placeholder the source system has used since 2014. Build that distribution check into the pipeline as an assertion, because the sentinel always shows up in a load long after the join was written.

Common mistake: Calling it generic skew and salting the key, which spreads NULLs that still cannot match anything.

Follow-up they will ask: How do you find a sentinel value acting like a null without knowing it is there?

Lesson · Simulation

senior

AQE skew join handling is enabled and a skewed task is still running an hour behind. Why might it not have fired?

Join strategies in Spark · tap to open the answer

Say this first: Skew splitting fires only when a partition clears both the median-multiple and the absolute-size thresholds, on a join shape it is allowed to split.

Remember these

  • Both skewedPartitionFactor and skewedPartitionThresholdInBytes must be exceeded
  • Uniformly large partitions are not skewed relative to the median
  • The split side must be replicable, which some outer join shapes forbid
  • AQE skew handling does nothing for skew in an aggregation

Why: AQE classifies a post-shuffle partition as skewed only if it is larger than spark.sql.adaptive.skewJoin.skewedPartitionFactor times the median partition size and larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes. Both conditions must hold, so a uniformly fat shuffle where every partition is huge has no skew relative to the median and is left alone. Beyond the thresholds, it needs spark.sql.adaptive.enabled and skewJoin.enabled, it only applies to join shapes it can split — splitting one side requires replicating the matching partition of the other, which outer join semantics can forbid — and it cannot help at all if the skew is in a groupBy rather than a join. The confirmation to look for is the word 'skewed' on the AQE shuffle read node in the SQL tab, or a post-shuffle task count higher than the configured partition count.

spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256MB")

Senior depth: Salt the hot key manually: append a random bucket suffix to the key on the large side, explode the small side across the same bucket range, join on the composite key, and drop the salt afterwards. It is more code and it changes the join key, so it needs a test that proves row counts and sums are unchanged. Reach for it only after confirming the key really is a single hot value rather than a wide fat tail.

Common mistake: Treating 'AQE is on' as proof that skew is handled, without checking the plan for a skewed shuffle read.

Follow-up they will ask: What do you do when AQE legitimately cannot split it?

Lesson · Simulation

architect

Two 4 TB fact tables are joined on customer_id every hour. Design the storage so the join stops shuffling.

Join strategies in Spark · tap to open the answer

Say this first: Bucket both tables on customer_id with the same bucket count and sort within buckets, so the join reads pre-partitioned data and skips the Exchange.

Remember these

  • Matching bucket column and count lets SortMergeJoin run with no Exchange
  • sortBy at write time lets the merge skip its sort too
  • Bucketing needs metastore tables; path-based reads do not carry the metadata
  • Bucketing does not fix skew — a hot key is still one hot bucket

Why: Bucketing writes each table into a fixed number of hash buckets on the join key, recorded in the table metadata. When both sides are bucketed on the same column with the same bucket count, the planner knows the required distribution is already satisfied and the SortMergeJoin appears with no Exchange underneath — you paid the shuffle once at write time instead of once an hour. Sorting within buckets at write time also lets the merge skip its sort. The costs are real: bucketing requires metastore-managed tables written through saveAsTable rather than raw paths, the bucket count is effectively permanent because changing it means rewriting both tables, writes get more expensive, and a hot customer_id is still a hot bucket so skew survives untouched. Spark can also join tables whose bucket counts differ by a multiple when spark.sql.bucketing.coalesceBucketsInJoin.enabled is set, which buys back some flexibility.

(fact.write
   .bucketBy(512, "customer_id")
   .sortBy("customer_id")
   .mode("overwrite")
   .saveAsTable("curated.fact_bucketed"))

Senior depth: Accept the shuffle and shrink what gets shuffled: pre-aggregate or narrow one side so the Exchange moves gigabytes rather than terabytes, cluster on the join key so scans prune hard, and consider whether an hourly incremental merge into a maintained wide table replaces the join entirely. Re-joining two full 4 TB facts every hour is usually a pipeline-shape problem, not a join-strategy problem.

Common mistake: Pitching bucketing as a free win without mentioning that the bucket count is a long-term commitment.

Follow-up they will ask: What would you choose instead on a Delta lakehouse where bucketing is awkward?

Lesson · Simulation

senior

Cached DataFrame, then an executor dies. What happens?

Cache and persist · tap to open the answer

Say this first: Lost partitions recompute from lineage — or fail if checkpoint wasn't used and shuffle files are gone.

Why: Cache is not a durable store. Storage tab shows cached partitions per executor. Executor loss → recompute. That's why checkpoint or a Delta write is the durable form.

Senior depth: Storage tab + stage that skips the scan. If the scan is back, cache missed or was evicted.

Common mistake: Treating cache as a table.

Follow-up they will ask: How do you see cache hit vs recompute in the UI?

Lesson · Simulation

advanced

You cached a DataFrame, then wrote a new query with an extra filter. Does it hit the cache?

Cache and persist · tap to open the answer

Say this first: Only if the new query's plan contains the cached plan as a subtree — Spark matches on the analyzed logical plan, not on the table name.

Remember these

  • Cache matching is by analyzed logical plan subtree, not by name
  • InMemoryTableScan in the plan proves the hit; FileScan proves the miss
  • Build from the cached DataFrame object to guarantee the match
  • A cache hit forfeits predicate pushdown into the file format

Why: When you cache, Spark records the analyzed logical plan of that DataFrame in the cache manager. On every subsequent query it tries to substitute an InMemoryRelation wherever it finds that exact plan as a subtree. Building on the cached DataFrame — cached_df.filter(...) — matches, so the plan shows InMemoryTableScan with the filter applied above it. Re-reading the source and writing what looks like the same transformations usually does not match, because the plans must be equivalent after analysis and small differences in projection order, aliases, or added columns are enough to miss. The way to confirm is explain('formatted'): InMemoryTableScan means a hit, a FileScan back on the source means a miss. Also note what you give up on a hit — the filter now runs over cached rows instead of being pushed into the Parquet scan, so a highly selective predicate can be faster uncached.

gold.cache(); gold.count()
gold.where("region = 'EU'").explain("formatted")   # look for InMemoryTableScan

Senior depth: Partly. The in-memory columnar format stores columns separately, so an InMemoryTableScan reads only the requested ones and can skip batches using per-batch statistics. What you cannot recover is the pruning you gave up at cache time: whatever columns were in the DataFrame when you cached it are what got materialized. Cache a narrow projection, not the whole table, and that problem disappears.

Common mistake: Assuming any query that mentions the same table reuses the cache.

Follow-up they will ask: Does a cached DataFrame still get column pruning when you select two of fifty columns?

Lesson · Simulation

senior

A 40-iteration algorithm gets slower every iteration even though you cache after each step. What is growing?

Cache and persist · tap to open the answer

Say this first: The logical plan — caching stores the data but does not truncate lineage, so planning cost grows with every iteration.

Remember these

  • cache stores data but keeps lineage; the plan keeps growing
  • Symptom is rising driver time between stages, not slower tasks
  • checkpoint writes to reliable storage and truncates the lineage
  • localCheckpoint truncates using executor disk and gives up recoverability

Why: Each iteration stacks another layer of transformations onto the previous plan, and cache does not cut that chain: it adds an InMemoryRelation node but keeps the full lineage underneath so lost partitions can be recomputed. By iteration 30 the driver is analyzing and optimizing a plan with thousands of nodes before any task is submitted, so you see driver CPU rising and the gap between cells growing while executor time stays flat. checkpoint() is the operator that actually cuts it: it writes the DataFrame to the directory set by sparkContext.setCheckpointDir and replaces the lineage with a read of that file, so the next iteration starts from a one-node plan. localCheckpoint() does the same truncation using executor disk instead of reliable storage, which is faster but loses the data — and therefore the ability to recover — if an executor dies. A good pattern for long iterative jobs is to cache every iteration and checkpoint every few.

spark.sparkContext.setCheckpointDir("/mnt/chk/iterative")
for i in range(40):
    df = step(df).cache()
    if i % 5 == 0:
        df = df.checkpoint()

Senior depth: Compare the wall clock of the cell against the summed stage durations in the Spark UI for the jobs it submitted. If the cell took ninety seconds and the stages account for twelve, the remaining time is driver-side analysis, optimization, and code generation. A driver thread dump during the gap will show the optimizer rules on the stack.

Common mistake: Adding executors when the time is being spent on the driver building a plan.

Follow-up they will ask: How would you prove the time is in planning rather than execution?

Lesson · Simulation

senior

Give me three concrete ways adding cache() can make a job slower.

Cache and persist · tap to open the answer

Say this first: It costs a materialization pass, it steals memory from execution, and it blocks pushdown into the source format.

Remember these

  • Materializing the cache is a cost paid even if reuse never happens
  • Storage memory and execution memory compete, so caching can cause shuffle spill
  • Reading from cache gives up predicate and column pushdown into Parquet
  • Cache narrow and filtered frames with real reuse, then unpersist

Why: First, populating the cache is real work: the partitions must be built and written into the block manager, and with a serialized storage level they are serialized and compressed too. If the frame is read once, that is pure overhead. Second, cached blocks live in storage memory in the same unified region as execution memory, so a heavy shuffle in the same job now has less room and starts spilling — the slowdown appears as Spill (Disk) in a stage that has nothing to do with the cache. Third, once a query reads from InMemoryTableScan, there is no Parquet scan left to push a filter into, so a selective predicate that previously skipped most row groups now scans every cached row. The rule is to cache a narrow, filtered frame that two or more downstream branches genuinely reuse, and to unpersist it as soon as the reuse is finished.

candidates = events.where("event_date >= '2026-10-01'").select("user_id", "event_type")
candidates.cache(); candidates.count()
# ... two branches read it ...
candidates.unpersist()

Senior depth: When the reuse spans jobs, sessions, or teams, or when the frame is large enough that holding it in memory squeezes execution. A Delta table is durable, readable by other jobs, carries statistics for pruning, and survives executor loss — all things a cache does not do. Cache is a within-session optimization; a table is a contract.

Common mistake: Caching every intermediate 'to be safe' and then blaming the shuffle spill it caused on executor memory.

Follow-up they will ask: When would you write an intermediate table instead of caching at all?

Lesson · Simulation

senior

Your predicate is on a non-partition column and the job still only touches 3% of the files. What made that work, and when does it stop working?

File formats: why Parquet reads less · tap to open the answer

Say this first: Parquet row-group min/max statistics let Spark skip groups whose value range cannot match the predicate.

Why: Every row group stores per-column min/max; with pushdown the reader drops groups outside the predicate range, which shows up as a small Input Size / Records relative to the table. It only works when data is physically clustered on that column — if values are scattered, every row group's min/max spans the whole domain and nothing is pruned.

Senior depth: Sort or cluster on the filter column at write time (Z-ORDER or liquid clustering on Delta). Statistics are only as useful as the physical layout — this is a layout decision, not a config flag.

Common mistake: Calling row-group skipping 'an index' and expecting it to work on randomly ordered data.

Follow-up they will ask: How do you make the data cooperate?

Lesson · Simulation

architect

The team wants gzip because it compresses the smallest. What do you tell them?

File formats: why Parquet reads less · tap to open the answer

Say this first: A gzip text file is not splittable, so one file becomes one task no matter how large it is.

Why: Gzip must be decompressed from the beginning, so Spark cannot split a gzip CSV or JSON file — a 20 GB file is a single task and the rest of the cluster idles. Snappy and zstd inside Parquet are applied per page and row group, so the file stays splittable; zstd trades some CPU for noticeably smaller files than snappy.

Senior depth: It is the opposite failure: thousands of tiny Parquet files make per-file open and metadata cost dominate, with the task duration histogram piled near zero. Compact on write or with OPTIMIZE and aim for files in the hundreds of MB.

Common mistake: Comparing codecs only on compressed size and ignoring splittability and decode cost.

Follow-up they will ask: Where does the small-file problem fit into this conversation?

Lesson · Simulation

advanced

Your filter is where(year(ts) = 2026) and PushedFilters comes back empty. Why, and what do you write instead?

File formats: why Parquet reads less · tap to open the answer

Say this first: A predicate on a function of a column is not a column-to-literal comparison, so it cannot be pushed into the file reader.

Remember these

  • Pushdown needs a bare column compared to a literal
  • year(), cast(), upper() around the column all block it
  • Rewrite as a half-open range on the raw column
  • PushedFilters and PartitionFilters in explain('formatted') are the proof

Why: Filter pushdown translates a Catalyst predicate into a format-level filter, and the Parquet reader can only evaluate simple comparisons between a column reference and a literal against the footer statistics. year(ts) = 2026 wraps the column in an expression, so there is nothing to compare to the min/max of ts and the predicate stays as a Filter node above the scan — every row group is read, decoded, and then discarded. Rewriting it as a half-open range on the raw column, ts >= '2026-01-01' and ts < '2027-01-01', gives the reader exactly the comparison it can use, and Input Size / Records on the FileScan drops accordingly. The same trap catches cast() on a partition column, upper() on a string key, and any predicate buried inside a UDF. Confirm with explain('formatted') and read the PushedFilters and PartitionFilters lines directly rather than assuming.

df.where("year(ts) = 2026").explain("formatted")                      # PushedFilters: []
df.where("ts >= '2026-01-01' AND ts < '2027-01-01'").explain("formatted")  # pushed

Senior depth: Physical clustering. Row-group skipping works by comparing the predicate against each group's min and max, so if the values are scattered across the file every group's range spans the whole domain and nothing can be skipped. Sorting on write, Z-ORDER, or liquid clustering on the filter column is what makes the statistics selective. Statistics are only as good as the layout they describe.

Common mistake: Assuming the optimizer rewrites year(ts) = 2026 into a range automatically for every type and source.

Follow-up they will ask: Your predicate is pushed but Input Size barely moved. What else is required?

Lesson · Simulation

senior

Give me the decision rule for Parquet versus ORC versus Avro — not a feature list.

File formats: why Parquet reads less · tap to open the answer

Say this first: Columnar for analytical reads of a few columns, row-based for whole-record writes and transport; everything else is ecosystem preference.

Remember these

  • Read pattern decides: few columns of many rows means columnar
  • Avro is row-based, which suits per-record writes and transport
  • ORC and Parquet differ mostly in ecosystem, not in capability class
  • The real mistake is letting the ingestion format become the analytics format

Why: The rule is the access pattern. Analytical queries touch a few columns of many rows, so a columnar layout — Parquet or ORC — lets the reader fetch only the needed column chunks and skip row groups on statistics. Ingestion, message transport, and replay touch whole records one at a time, which is where a row-based format like Avro wins: appending one record does not require buffering a row group, and Avro's schema-with-the-data model fits a Kafka pipeline's schema-evolution story. Between the two columnar options the technical differences are real but secondary — ORC carries stride-level indexes and optional bloom filters and is native to the Hive lineage, while Parquet has broader engine and tooling support — so the honest answer is to pick the one your table format and query engines are built around and stop arguing. The failure mode to call out is using a row format for the analytics layer because that is what ingestion produced.

Senior depth: One level above. Delta and Iceberg store data as Parquet or ORC and add a transaction log on top, which gives atomic commits, time travel, schema evolution, and file-level statistics for pruning without listing. So the question is not Delta versus Parquet — it is which file format Delta writes, and whether you want a log above it. For anything a pipeline writes more than once, you want the log.

Common mistake: Reciting benchmark comparisons instead of naming the access pattern that makes the choice.

Follow-up they will ask: Where does a table format like Delta or Iceberg sit in this comparison?

Lesson · Simulation

architect

A 3 TB table is stored in roughly 40 KB files and every consumer is complaining. Design the fix and the guardrail that stops it recurring.

File formats: why Parquet reads less · tap to open the answer

Say this first: Compact to files in the hundreds of MB, fix the writer that produced fragments, and monitor average file size as an SLO.

Remember these

  • Tens of millions of objects means listing and task launch dominate real work
  • Compaction fixes today; the writer is what fixes tomorrow
  • Trigger interval and partition-column cardinality are the usual root causes
  • Track average file size per table as a monitored SLO

Why: At 40 KB per file a 3 TB table is tens of millions of objects, and every read pays listing, footer opens, and task launch per object — the work is all overhead and almost none of it is data. The immediate remedy is compaction: OPTIMIZE on Delta, or a rewrite job that reads and writes back with a sane partition count. That is a one-time fix, so the durable part is the writer: match the write's partition count to the volume per batch, bound file size with maxRecordsPerFile, lengthen the streaming trigger interval so each micro-batch emits fewer files, and avoid partitioning on a column with enough cardinality to guarantee tiny directories. The guardrail is a metric: record file count and average file size per table after every write and alert when the average drops below a threshold, because this problem always arrives gradually and is only noticed when a consumer's SLA breaks.

-- one-time remediation
OPTIMIZE curated.events;
-- and in the writer
SET spark.databricks.delta.optimizeWrite.enabled = true;

Senior depth: Make it incremental and bounded. Compact only the partitions written since the last run rather than the whole table, run it on its own small cluster off the critical path, and cap how much it rewrites per execution so it never becomes a surprise multi-hour job. Full-table compaction on a shared cluster during business hours is how a fix becomes the next incident.

Common mistake: Scheduling compaction and calling it done, while the writer keeps producing fragments every hour.

Follow-up they will ask: Compaction competes with your pipelines for the same cluster. How do you schedule it?

Lesson · Simulation

senior

Your window job has one task running for 40 minutes while 199 finished in seconds. What did you write?

Window functions: keep the row, add the answer · tap to open the answer

Say this first: A Window.orderBy(...) with no partitionBy — every row was shuffled into a single partition.

Why: Without partitionBy, Spark must order the whole dataset globally, so it plans an Exchange SinglePartition beneath WindowExec and one task processes everything. The stage shows exactly 1 task carrying the entire Shuffle Read plus a large Spill (Disk) while the rest of the cluster idles.

Senior depth: Reduce first, then order: filter or pre-aggregate to candidates, or rank within real partitions and stitch them with a small per-partition offset table. A truly global ordering over billions of rows is a design problem, not a tuning knob.

Common mistake: Diagnosing it as skew and salting a key that the window does not even partition by.

Follow-up they will ask: How do you compute a global row number without a single-task stage?

Lesson · Simulation

architect

Dedup with row_number() = 1 versus dropDuplicates — which goes in the pipeline and why?

Window functions: keep the row, add the answer · tap to open the answer

Say this first: row_number() when you must keep a specific survivor; dropDuplicates when any row per key is acceptable.

Why: dropDuplicates(keys) dedups via a shuffle and aggregate and hands you an arbitrary row per key. partitionBy(keys).orderBy(updated_at.desc) with a filter on row_number() = 1 pays for a Sort inside WindowExec but deterministically keeps the newest record — which is exactly what CDC and SCD merges require.

Senior depth: Add a deterministic tiebreaker such as a sequence number or commit ordering. Without one, reruns pick different survivors and downstream diffs look like data loss.

Common mistake: Running dropDuplicates on a CDC feed and shipping a random version of each record.

Follow-up they will ask: What breaks if the ordering column has ties?

Lesson · Simulation

advanced

One select has three window functions over three different WindowSpecs. How many shuffles does that plan contain?

Window functions: keep the row, add the answer · tap to open the answer

Say this first: One Exchange per distinct partitioning — functions sharing a partitionBy collapse into one WindowExec, different partitionBy columns each force their own.

Remember these

  • Windows with identical specs are fused into one WindowExec
  • Same partitionBy, different orderBy shares the Exchange but adds a Sort
  • A different partitionBy column means a whole new shuffle
  • Count Exchange nodes under Window operators in explain('formatted')

Why: The planner groups window expressions by their window specification. Functions with the same partitionBy and the same orderBy become a single WindowExec over one Exchange hashpartitioning and one Sort. Functions that share the partitionBy but differ in orderBy still share the Exchange, since the distribution requirement is identical, but each needs its own Sort and its own Window operator stacked on top. Functions partitioned by a different column cannot reuse any of it: the planner inserts another Exchange, another Sort, and another Window, and you now pay a second full shuffle of the dataset. So three specs over three different keys is three shuffles, which is why adding one 'small extra metric' partitioned by a different column can double a job's runtime. Read it off explain('formatted') by counting Exchange nodes beneath the Window operators.

w1 = Window.partitionBy("customer_id").orderBy("ts")
w2 = Window.partitionBy("customer_id").orderBy(F.desc("amount"))  # same Exchange
w3 = Window.partitionBy("region").orderBy("ts")                    # second Exchange

Senior depth: Compute them in separate passes at their natural grain and join the small result back, rather than forcing both windows over the full detail. The region metric is usually a per-region aggregate, which is a groupBy producing a tiny frame you can broadcast onto the detail. Paying a broadcast join instead of a full second shuffle of the fact table is nearly always the cheaper plan.

Common mistake: Assuming all window functions in one select share a single shuffle regardless of their keys.

Follow-up they will ask: You need metrics at customer grain and region grain in one output. How do you avoid the second shuffle?

Lesson · Simulation

advanced

Someone used row_number() to take the top 3 products per category and the results are unstable across runs. Walk through rank, dense_rank, and row_number.

Window functions: keep the row, add the answer · tap to open the answer

Say this first: row_number breaks ties arbitrarily, rank gives ties the same number and skips the next, dense_rank gives ties the same number and does not skip.

Remember these

  • row_number is 1,2,3; rank is 1,1,3; dense_rank is 1,1,2
  • row_number with ties needs a deterministic tiebreaker or it is unstable
  • Top-N including ties means rank or dense_rank, not row_number
  • The rank filter sits above the window, though newer Spark can push a group limit down

Why: With sales of 100, 100, 90, row_number assigns 1, 2, 3 — the two hundreds are separated in whatever order the sort happened to produce, which is not stable across runs because the sort is not total. rank assigns 1, 1, 3, preserving the tie and skipping position 2. dense_rank assigns 1, 1, 2. For 'top 3 including ties' you want rank() <= 3 or dense_rank() <= 3 depending on whether skipped positions should consume slots; for exactly three rows you want row_number, but then you must add a deterministic tiebreaker column to the orderBy or accept nondeterminism. Note also that the filter on the rank is applied above WindowExec, so conceptually the whole partition is still ordered — though recent Spark versions can push a per-group limit beneath the window for rank-style functions, which you may see as a WindowGroupLimit node in the plan.

w = Window.partitionBy("category").orderBy(F.desc("sales"), F.asc("product_id"))
top3 = df.withColumn("rn", F.row_number().over(w)).where("rn <= 3").drop("rn")

Senior depth: Parallelism collapses to eight. partitionBy is the shuffle key, so eight distinct categories means eight non-empty post-shuffle partitions and eight tasks, each holding an eighth of a billion rows to sort. The Stage page shows a handful of enormous tasks with heavy spill. Pre-aggregate to the grain the ranking actually needs — product totals per category — before ranking, so the window runs over thousands of rows instead of a billion.

Common mistake: Using row_number for 'top N' and silently dropping legitimate ties every run.

Follow-up they will ask: The category column has only eight distinct values on a billion-row table. What goes wrong?

Lesson · Simulation

senior

Your window stage spills hundreds of GB, but the partitionBy key is evenly distributed. What else causes that?

Window functions: keep the row, add the answer · tap to open the answer

Say this first: WindowExec buffers rows of the current frame in memory, so wide rows and unbounded frames spill even without skew.

Remember these

  • WindowExec buffers whole rows of the active frame, not just referenced columns
  • Unbounded frames make the buffer grow with the partition
  • windowExec.buffer.in.memory.threshold controls when spilling starts
  • Narrow the projection before the window instead of raising thresholds

Why: After the Exchange and Sort, WindowExec walks the sorted partition and keeps the rows of the active frame buffered so the aggregate can be evaluated. With a frame of unbounded preceding to current row, that buffer grows with the partition, and every column of every buffered row is held — not just the ones the window function references. So a partition of a few million wide rows will spill even when the key distribution is perfectly flat. Two configs control when it gives up on memory: spark.sql.windowExec.buffer.in.memory.threshold sets how many rows are held before spilling begins, and spark.sql.windowExec.buffer.spill.threshold bounds it further. The better fix is to shrink the rows rather than raise the thresholds: select only the columns the window needs, compute the window, then join the result back onto the detail by key if you need the wide columns in the output.

keys = df.select("customer_id", "ts", "amount")
ranked = keys.withColumn("running", F.sum("amount").over(w))
out = df.join(ranked.select("customer_id", "ts", "running"), ["customer_id", "ts"])

Senior depth: Yes, materially. A bounded frame means the buffer holds at most the frame width, so memory per partition becomes roughly constant instead of growing with partition size. If the metric is genuinely a trailing seven-row or seven-day window, say so explicitly rather than relying on the default running-total frame — it is both more correct and far cheaper.

Common mistake: Raising executor memory when the row width going into the window is what needs to change.

Follow-up they will ask: Would a bounded frame like rowsBetween(-6, 0) change the spill behaviour?

Lesson · Simulation

senior

Walk me up the fix ladder for a row-at-a-time Python UDF, cheapest first.

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: Built-in expression, then higher-order functions, then pandas_udf, and only then a plain Python UDF.

Why: Built-ins keep codegen and pushdown. Higher-order functions (transform, filter, exists, aggregate) handle array and map logic without leaving SQL. pandas_udf ships columnar Arrow batches, so you pay serialization per batch instead of per row and get vectorized pandas or NumPy code. A row-wise Python UDF is the last resort.

Senior depth: It is still opaque to Catalyst, so nothing pushes through it, and on Databricks it still knocks the query off Photon. Faster, not free.

Common mistake: Jumping straight to pandas_udf when composing two built-ins would have done it.

Follow-up they will ask: What does a pandas UDF still not fix?

Lesson · Simulation

architect

A Scala UDF runs inside the JVM, so is it safe to use anywhere?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It skips the Python round trip, but it is still a black box to the optimizer.

Why: A Scala or Java UDF executes in the executor JVM with no cross-process serialization, so throughput is far better than Python. Catalyst still cannot rewrite or push it, it can break whole-stage codegen for the operators around it, and null handling is entirely your responsibility. A native Expression or a composition of built-ins is still preferable when the logic can be expressed.

Senior depth: Rerun the query with that column replaced by a literal. If runtime collapses, it is the UDF; if not, you were about to optimize the wrong operator.

Common mistake: Treating 'we made it a Scala UDF' as equivalent to a native expression in a design review.

Follow-up they will ask: How do you prove the UDF is or is not your bottleneck?

Lesson · Simulation

advanced

A UDF calls an external REST API once per row. Beyond speed, what is wrong with that, and how do you restructure it?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It is nondeterministic and retried unpredictably, and it fans out uncontrolled concurrency at the remote service — batch the calls and bound the parallelism.

Remember these

  • Spark may evaluate an expression more than once; side effects are not safe
  • Task count becomes your concurrency against the remote service
  • Batch with mapInPandas and set the partition count deliberately
  • Materializing the lookup into a table turns the call into a broadcast join

Why: Spark assumes expressions are pure and may evaluate them more than once: a task retry, speculative execution, or an optimizer rewrite can all cause a second call, so duplicate side effects and inconsistent results are structural rather than unlucky. Concurrency is the second problem — every task runs the UDF simultaneously, so a 400-task stage means up to 400 concurrent callers hitting a service that was sized for far less, and your own job causes the rate limiting it then fails on. The restructure is to stop working per row: use mapInPandas or a pandas_udf so each call handles a batch, control the fan-out by setting the partition count deliberately rather than inheriting it, and add retry and backoff inside the function because Spark's task-level retry is far too coarse. The better answer when it is available is to not call the service in the data path at all — materialize the lookup into a table on a schedule and turn the enrichment into a broadcast join.

def enrich(batches):
    for pdf in batches:
        pdf["geo"] = lookup_many(pdf["ip"].tolist())   # one call per batch
        yield pdf

df.repartition(32).mapInPandas(enrich, schema=out_schema)

Senior depth: Handle the failure inside the function rather than letting it fail the task, since a task failure reprocesses the entire partition and re-issues every call in it. Catch per-item errors, emit a null or an error column for the failed items, and let a downstream step reprocess that subset. The design principle is to make the retry granularity match the call granularity.

Common mistake: Fixing the slowness by increasing executor count, which multiplies the load on the service that was already the bottleneck.

Follow-up they will ask: How would you make the batched version resilient to a partial failure inside one batch?

Lesson · Simulation

senior

A Python UDF is declared to return IntegerType and about 2% of its output comes back null, with no errors anywhere. Explain.

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: When a returned value does not fit the declared return type, PySpark yields null instead of raising.

Remember these

  • A return value that does not match the declared type becomes null, not an error
  • Scala UDFs with primitive parameters turn input nulls into 0 or false
  • Diagnose by selecting rows where output is null and input is not
  • Use nullable types and explicit edge-case handling inside the function

Why: The return type you pass to udf() is a contract Spark applies to whatever Python hands back. If the function returns a float, a string, or a value outside the integer range for those 2% of rows, the conversion fails and the result is null rather than an exception — silent data loss that looks like upstream nulls. The same category of trap exists on the input side for JVM UDFs: a Scala UDF with primitive parameters receives 0 or false where the column was null, because a primitive cannot hold null, so your logic sees a value that was never in the data. The diagnosis is to filter to the rows where the output is null and the input is not, then run the function on those inputs in plain Python to see what it actually produced. The fixes are to widen or correct the declared return type, to handle the edge cases explicitly inside the function, and to use boxed or Option types in Scala UDFs so nulls stay null.

bad = df.where(F.col("score").isNull() & F.col("raw").isNotNull())
bad.select("raw").show(20, truncate=False)

Senior depth: Assert on it. Add a post-transformation check that the null rate of the derived column is within an expected band and fail the job when it is not, and unit-test the Python function directly against the known edge cases rather than only testing the DataFrame end to end. Silent null production is the characteristic UDF failure, so the test has to be about null rate, not about the happy path.

Common mistake: Blaming the source data for nulls that the UDF's own type contract created.

Follow-up they will ask: How would you catch this class of bug before it reaches production?

Lesson · Simulation

senior

When does asNondeterministic() actually matter, and what does it not fix?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It stops Catalyst from reordering, pushing, or collapsing the UDF — but it does not make repeated evaluation produce the same answer.

Remember these

  • Catalyst assumes determinism and may reorder or duplicate the UDF
  • asNondeterministic pins it in place and prevents those rewrites
  • It does not make task retries reproduce the same values
  • For reproducibility, derive from a column seed or materialize the values

Why: By default Catalyst treats a UDF as deterministic and is therefore free to push it below or above a filter, duplicate it into both branches of a plan, or collapse adjacent projections that reference it. For a function involving randomness, the current time, or an external call, those rewrites change results — for example a random-sampling UDF pushed beneath a filter is evaluated on a different row set than you wrote. Marking it with asNondeterministic() tells the optimizer to leave it where it is and evaluate it once per row, which restores the semantics you intended. What it does not do is guarantee stability across retries: if a task fails and reruns, the function runs again and produces different values, and any already-committed downstream output can disagree. For reproducibility you need the nondeterminism removed from the data path — seed it from a column, or materialize the generated values into a table and join against them.

import random
from pyspark.sql.functions import udf
sample_flag = udf(lambda: random.random() < 0.01, "boolean").asNondeterministic()

Senior depth: rand() is marked nondeterministic in the optimizer but takes a seed, and the seed is combined with the partition index — so the same plan over the same partitioning gives the same values. That is reproducible only as long as the partitioning is stable, which it is not across a repartition or a change in shuffle partition count. If reproducibility genuinely matters, materialize the generated column once and read it back.

Common mistake: Reaching for asNondeterministic to fix inconsistent results across runs, which is a retry problem it cannot touch.

Follow-up they will ask: Where does Spark's own rand() sit on this spectrum?

Lesson · Simulation

senior

You have 200 executors and a 2 GB driver. Which jobs will still fail?

Driver vs executors · tap to open the answer

Say this first: Any action that pulls the full result to the driver — collect, toPandas, display of an unaggregated frame.

Why: Executor count does not grow driver heap. A 2 TB collect still targets one JVM. Broadcast join of a 8 GB dimension also hits the driver, then every executor.

Senior depth: Aggregate or sample on executors, write the rest, never pull the grain to the notebook.

Common mistake: Scaling workers to fix a driver OOM.

Follow-up they will ask: How would you rewrite the notebook so the driver stays small?

Lesson · Simulation

advanced

A job fails with 'Total size of serialized results is bigger than spark.driver.maxResultSize'. Is raising the limit the fix?

Driver vs executors · tap to open the answer

Say this first: Almost never — the setting is a guardrail against pulling a dataset to one JVM, and the real fix is to stop pulling it.

Remember these

  • The limit applies to the sum of all task results in one action
  • Small results come inline; large ones arrive via the block manager
  • Raising it trades a clear abort for a driver heap OOM
  • Setting it to 0 removes the guardrail, not the problem

Why: maxResultSize bounds the sum of serialised results across all tasks of a single action, so it fires on the aggregate, not on any one task. Small task results come back inline over the RPC layer, bounded by spark.rpc.message.maxSize; larger ones are stored as blocks and fetched by the driver through the block manager, which is why the driver's memory grows while the error talks about serialised bytes. Raising the limit converts a clean, early abort into a driver heap OOM or a GC death spiral somewhere further along. The cases where raising it is legitimate are narrow: a genuinely needed result in the low gigabytes with a driver node sized for several times that. Otherwise restructure the action — aggregate on executors, write to a table and read it back, or take a bounded sample. Setting it to 0 disables the check entirely and is the fastest way to turn a readable error into an unexplained kernel death.

# instead of collecting every key
keys = spark.sql('SELECT key FROM t GROUP BY key')
keys.write.mode('overwrite').saveAsTable('scratch.keys')

Senior depth: Treat maxResultSize as a design assertion: 'no action returns more than this much to one JVM.' When it fires, the right artefact is a changed query, not a changed config. The only time I raise it is for a known, bounded export, and then I raise the driver node size in the same change.

Common mistake: Bumping the value until the error stops, then reporting the driver OOM as a separate unrelated incident.

Follow-up they will ask: Which actions besides collect can trip this limit?

Lesson · Simulation

senior

A streaming job's driver heap creeps upward over six days and then dies. Executors are healthy the whole time. What is accumulating?

Driver vs executors · tap to open the answer

Say this first: Driver-side bookkeeping — retained UI state, query progress history, and a backlogged listener queue — not your data.

Remember these

  • Retained Jobs/Stages/Tasks and SQL executions are driver-side memory
  • A short trigger interval creates executions far faster than batch does
  • A slow listener backs up the event queue and grows the heap
  • Monotonic driver growth with flat executors is a bookkeeping leak

Why: The driver keeps in-memory history for the UI: completed jobs, stages, and tasks bounded by spark.ui.retainedJobs, spark.ui.retainedStages, and spark.ui.retainedTasks, plus SQL executions bounded by spark.sql.ui.retainedExecutions. A micro-batch stream produces a query every trigger interval, so at a ten-second trigger you create tens of thousands of executions a day and the default retention is sized for batch jobs, not for that rate. Events reach those listeners through an async queue sized by spark.scheduler.listenerbus.eventqueue.capacity; when a listener is slow, the queue backs up, memory grows, and you start seeing dropped-event warnings in the driver log. Streaming query progress objects and accumulator state add to the same pile, and the ContextCleaner only reclaims broadcasts and shuffles once references are gone. The signature is unmistakable: a sawtooth-free, monotonic heap climb with flat executor memory and no change in input volume. Lower retention, trim custom listeners, and restart on a schedule until the growth is flat.

spark.conf.set('spark.ui.retainedJobs', '200')
spark.conf.set('spark.ui.retainedStages', '200')
spark.conf.set('spark.sql.ui.retainedExecutions', '200')

Senior depth: Long-running drivers need a retention budget the same way executors need a memory budget. Any custom QueryExecutionListener or StreamingQueryListener you add is driver code on the hot path — if it writes to an external system synchronously, it will back the event queue up, and the symptom will look like a memory leak rather than a slow HTTP call.

Common mistake: Hunting for a data leak in the streaming logic when no executor metric has moved.

Follow-up they will ask: How would you tell a retention leak from a genuine state-store growth problem?

Lesson · Simulation

senior

Dynamic allocation scaled the job to 300 executors and it got slower. Explain the mechanisms.

Driver vs executors · tap to open the answer

Say this first: More executors multiply shuffle fetch connections, broadcast copies, and decommission-driven recomputation, none of which the scan time pays back.

Remember these

  • Reducer fetch fan-in grows linearly with executor count
  • Broadcasts cost one copy per executor, so wide clusters multiply them
  • Losing an executor loses its shuffle files unless tracking is enabled
  • FetchFailed forces a map-stage retry, undoing completed work

Why: A reducer fetches from every map-output location, so going from 30 to 300 executors turns each reducer's fetch fan-in into ten times as many connections, and Shuffle Read Blocked Time starts dominating task duration. Every broadcast is materialised once per executor, so a 400 MB dimension that cost 12 GB cluster-wide now costs 120 GB and the broadcast step itself takes longer to distribute. Scale-down is the sharper edge: without spark.dynamicAllocation.shuffleTracking.enabled or an external shuffle service, removing an executor destroys the shuffle files it produced, and the next reducer that needs them hits FetchFailed, which retries the whole map stage. That produces the classic pattern of a job that oscillates between scaling up and recomputing stages it had already finished. Bounding spark.dynamicAllocation.maxExecutors to something matched to the stage's real partition count usually restores the previous runtime immediately.

spark.conf.set('spark.dynamicAllocation.maxExecutors', '60')
spark.conf.set('spark.dynamicAllocation.shuffleTracking.enabled', 'true')

Senior depth: Parallelism is capped by partitions, not by executors. Once executor cores exceed the stage's partition count, extra executors add shuffle topology and broadcast cost while sitting idle. I cap maxExecutors from the widest stage's partition count and treat anything above that as a cost with no upside.

Common mistake: Treating executor count as monotonically helpful and reading repeated stage retries as flaky infrastructure.

Follow-up they will ask: How many executors can a stage with 200 partitions actually use?

Lesson · Simulation

senior

How do you explain a 1:N:M relationship on a whiteboard?

Jobs, stages, and tasks · tap to open the answer

Say this first: 1 action → N stages (shuffles + 1) → M tasks per stage (partitions).

Why: Draw: show() → Job 1 → Stage 0 scan/filter → Exchange → Stage 1 aggregate → few rows to driver. Tasks = partitions in each stage, possibly different counts after shuffle.

Common mistake: Drawing one box called 'the cluster'.

Follow-up they will ask: What changes M without changing N?

Lesson · Simulation

advanced

Six narrow transformations in a row become one task's work. Explain what happens inside that task and where whole-stage codegen fits.

Jobs, stages, and tasks · tap to open the answer

Say this first: The operators are fused into a single generated loop over the partition's rows, so no intermediate collection is ever materialized.

Remember these

  • Narrow operators in one stage run as one fused loop per partition
  • WholeStageCodegen in the plan marks what got fused, with an id per region
  • Fusion removes per-row virtual calls and intermediate row objects
  • Python UDFs and some operators break the chain into separate codegen regions

Why: Narrow transformations have a one-to-one partition dependency, so the DAGScheduler places them in the same stage and each task runs the entire chain over its own partition. Catalyst then goes further with whole-stage code generation: it compiles the fused operators into a single Java method that pulls a row, applies every projection and filter in sequence, and emits it — which is why the plan shows a WholeStageCodegen node with an id wrapping several operators. The payoff is no virtual function call per operator per row and no intermediate row objects, so the chain costs roughly one pass over the data rather than six. Not everything can be fused: an operator without a codegen implementation, a Python UDF's BatchEvalPython, or a plan too large for the generated-method size limit will break the chain, and you see that as multiple separate WholeStageCodegen nodes or operators sitting outside one entirely. Those break points are worth reading, because an operator that falls out of codegen is often the expensive one.

df.select("id", "amount").where("amount > 0").withColumn("net", F.col("amount") * 0.9).explain("formatted")

Senior depth: Look for operators that are not wrapped in a WholeStageCodegen region, or for several small regions where you expected one. A sudden proliferation of regions after a code change usually points at a newly introduced operator that has no codegen path. You can also disable spark.sql.codegen.wholeStage temporarily to measure what fusion is buying, which is a useful diagnostic and a terrible production setting.

Common mistake: Thinking each transformation writes an intermediate dataset that the next one reads.

Follow-up they will ask: How would you tell from a plan that codegen was disabled or broke?

Lesson · Simulation

advanced

Tasks compute in 200 ms but Scheduler Delay is 8 seconds. What creates scheduler delay and what do you do about it?

Jobs, stages, and tasks · tap to open the answer

Say this first: It is the time spent getting the task to the executor and the result back — serialization, dispatch, and driver contention — and it means your tasks are too small or your closure is too big.

Remember these

  • Scheduler Delay is dispatch and result transport, not compute
  • Too many tiny tasks makes the driver the bottleneck
  • A large captured variable is serialized into every task
  • Broadcast large lookups so they ship once per executor

Why: Scheduler Delay on the Stage page is the task's wall clock minus the time actually spent executing and deserializing, so it covers the driver serializing and dispatching the task, the executor waiting for a free slot, and the result travelling back. Two causes dominate. One is sheer task count: a driver dispatching hundreds of thousands of tiny tasks becomes the bottleneck, and the remedy is fewer, larger partitions. The other is a fat task closure — a Python function or Scala lambda that captures a large lookup structure gets serialized into every single task, so a 50 MB capture across 20,000 tasks is a terabyte of dispatch. The fix there is to broadcast the structure once with sparkContext.broadcast and reference the broadcast variable, so it ships per executor rather than per task. Driver GC pressure produces the same symptom, so check driver memory and GC time before concluding anything.

lookup_bc = spark.sparkContext.broadcast(big_dict)
rate = F.udf(lambda k: lookup_bc.value.get(k, 1.0), "double")

Senior depth: Read Task Deserialization Time alongside Scheduler Delay in Summary Metrics. A large per-task deserialization time with a modest task count points at a fat closure; near-zero deserialization time with an enormous task count points at dispatch volume. The driver log also prints a warning when a task binary is unusually large, which is the cheapest confirmation available.

Common mistake: Adding executors, which increases the number of things the already-saturated driver has to coordinate.

Follow-up they will ask: How do you confirm the closure size is the problem rather than the task count?

Lesson · Simulation

senior

You wrote one groupBy and the job shows seven stages. Where did the other stages come from?

Jobs, stages, and tasks · tap to open the answer

Say this first: Every shuffle boundary, broadcast build, sort, and AQE re-plan adds stages that your code never names.

Remember these

  • Stages come from the physical plan, not from the number of wide calls in code
  • Each Exchange, broadcast build, and global sort contributes a boundary
  • AQE materializes query stages incrementally and can add more
  • Map plan nodes to stages in the SQL tab rather than counting operators

Why: Stage count is a property of the physical plan, not of how many wide operations you typed. Each Exchange cuts a boundary, so a join adds one or two, a window adds its own, and a global sort adds an exchange plus a sort stage. A BroadcastExchange is built in its own job and stage before the probe side runs. AQE re-plans between stages and can submit additional stages as it materializes query stages one at a time. Reading a source with schema inference submits a stage before your query starts, and on Delta or similar formats the commit and statistics work shows up too. The reliable way to account for it is to open the SQL tab, click the query, and walk the plan node by node mapping each Exchange and each broadcast to the stage it produced — that mapping is the skill, and it is also what tells you which of those seven stages is worth your attention.

Senior depth: Sort the stages by duration and start with the one that owns the largest share of wall clock, then read its Summary Metrics to decide whether it is a data-volume problem, a skew problem, or an overhead problem. Optimizing a stage that accounts for four percent of the runtime is the most common way to spend a sprint on nothing.

Common mistake: Counting groupBys and joins in the source code and asserting the stage count from that.

Follow-up they will ask: Of those seven, how do you pick the one to optimize?

Lesson · Simulation

senior

How do you use the DAG to pick a broadcast vs a shuffle without guessing?

The Spark DAG · tap to open the answer

Say this first: Find the join node and whether an Exchange sits above each child.

Why: No Exchange on the fact side means the small side was broadcast and the big side stays where it is. An Exchange above both children means a shuffle-based SortMergeJoin, so every row of both tables crosses the network. The SQL text tells you nothing here — the same JOIN compiles to either plan depending on the estimated size of the build side against spark.sql.autoBroadcastJoinThreshold. Read the plan, not the query. Under AQE the physical plan can also switch after the first shuffle reports real sizes, so the Spark UI's SQL tab is the version that actually ran.

Common mistake: Reading only the SQL string.

Follow-up they will ask: Where is that visible besides explain()?

Lesson · Simulation

advanced

For read → filter → withColumn → groupBy().agg() → write, how many stages do you expect and where exactly is the boundary?

The Spark DAG · tap to open the answer

Say this first: Two stages, cut at the Exchange the aggregation's shuffle introduces.

Remember these

  • Stage count = number of Exchange nodes + 1, for a linear plan
  • Narrow operators pipeline inside one task; they never add a stage
  • HashAggregate appears twice: partial before the shuffle, final after

Why: filter and withColumn are narrow: each output partition depends on exactly one input partition, so the scheduler pipelines them into the same task as the scan and whole-stage codegen fuses them into one generated loop. groupBy().agg() needs every row for a key in one place, so the physical plan inserts an Exchange hashpartitioning, which is a wide dependency and therefore a stage boundary. Stage one runs scan → filter → project → partial HashAggregate and writes shuffle files; stage two reads them, runs the final HashAggregate, and writes. In explain("formatted") you can count stages by counting Exchange nodes and adding one. The write itself adds no stage unless it repartitions — a partitionBy that forces a sort on the partition columns does.

df.groupBy("region").sum("amount").explain("formatted")
# count the Exchange nodes to count the stage boundaries

Senior depth: State it as a rule and then qualify it. Exchange count plus one holds for a linear plan, but a join has two children, so a branching plan gives you parallel stages the scheduler can run concurrently once their parents finish. AQE can also add stages after the fact, so the count in the UI can exceed what a static explain predicted.

Common mistake: Counting one stage per transformation, as if every operator cost its own pass over the data.

Follow-up they will ask: What would adding a join to the same plan do to the stage count?

Lesson · Simulation

advanced

Is lineage the same thing as the DAG the DAGScheduler schedules?

The Spark DAG · tap to open the answer

Say this first: No — lineage is the dependency record used for recovery; the scheduled DAG is the stage graph built from the physical plan.

Remember these

  • Lineage exists for fault recovery; the stage DAG exists for scheduling
  • Catalyst sits between them and can rewrite the plan entirely
  • checkpoint() truncates lineage; cache() keeps the full parent chain

Why: Lineage is the chain of parent dependencies every RDD or DataFrame carries, and its purpose is recovery: lose a partition and Spark replays its ancestors to rebuild just that partition. The DAG the scheduler works with is downstream of Catalyst — it is the physical plan grouped into stages at wide dependencies, with one task per partition inside each stage. Catalyst can collapse, reorder, or delete operators from your lineage before the scheduler ever sees it, so the two graphs are not the same shape. The practical consequence is that a long lineage is a recovery and planning cost, while a deep stage graph is a shuffle cost. checkpoint() is the operation that truncates lineage, writing to reliable storage and cutting the parent chain, which is not what cache() does.

spark.sparkContext.setCheckpointDir("/tmp/chk")
df2 = df.checkpoint()  # materialized, parent lineage dropped

Senior depth: localCheckpoint() writes to executor local disk, so it cuts lineage but is not fault tolerant — lose the executor and the data is gone with no parents left to replay from. Use it inside an iterative loop on a stable cluster, never as the durability story for a long pipeline that has to survive a node failure.

Common mistake: Using cache() to 'cut lineage' — cache keeps every parent so it can recompute evicted blocks.

Follow-up they will ask: When is localCheckpoint() the wrong choice?

Lesson · Simulation

senior

A task fails with FetchFailedException. Does the DAGScheduler or the TaskScheduler handle that, and what happens next?

The Spark DAG · tap to open the answer

Say this first: The DAGScheduler, because a missing shuffle block means a whole parent stage has to be resubmitted, not just a task retried.

Remember these

  • TaskScheduler retries tasks; DAGScheduler retries whole stages
  • FetchFailedException means lost shuffle output, not a slow task
  • External shuffle service keeps shuffle files alive after executor loss

Why: The TaskScheduler owns individual task attempts: it launches them on executors, retries ordinary failures up to spark.task.maxFailures, and handles speculation. A FetchFailedException is different — it says the shuffle output a reduce task needs is gone, usually because the executor that wrote it died. The DAGScheduler unregisters that map output, marks the parent shuffle map stage as failed, and resubmits only its missing partitions before retrying the reduce stage. In the UI this looks like a stage number repeating with an attempt counter and a 'Resubmitted' note, and the job's wall clock jumps far more than one lost task would explain. Running an external shuffle service keeps shuffle files alive past executor death, which is why it matters so much under dynamic allocation.

Senior depth: Dynamic allocation removes idle executors, and if those executors still hold shuffle files the next reduce stage fetch-fails and the map stage is recomputed. Enable the external shuffle service, or on newer Spark set spark.dynamicAllocation.shuffleTracking.enabled, so the scheduler will not reclaim an executor whose shuffle output is still needed.

Common mistake: Raising spark.task.maxFailures to fix fetch failures — the task was never the problem.

Follow-up they will ask: Why does this get worse with dynamic allocation?

Lesson · Simulation

senior

Catalyst picked SMJ. Stats were wrong. What's your move?

Catalyst optimizer · tap to open the answer

Say this first: ANALYZE TABLE / fix stats, add a broadcast hint if you know the size, or enable AQE join conversion.

Why: Don't hint forever as a substitute for stats. Hints rot. Measure with actual size after filters — AQE sees runtime sizes.

Senior depth: When you've measured the build side after filters and documented why stats cannot see it (UDF, JDBC, etc.).

Common mistake: Broadcast hint on a table that will grow 20×.

Follow-up they will ask: When is a hint the right senior answer?

Lesson · Simulation

advanced

Your filter appears as a separate Filter node above the FileScan instead of inside PushedFilters. Give me three reasons that happens.

Catalyst optimizer · tap to open the answer

Say this first: The predicate is opaque to the source, the source cannot evaluate that expression, or only part of it was pushed and the residual stays above.

Remember these

  • A UDF or non-deterministic expression can never be a PushedFilter
  • An implicit cast on the column side can block the match
  • PartitionFilters skip directories; PushedFilters skip row groups
  • Input Size / Records on the scan is the proof, not the plan text

Why: First, the predicate may not be expressible at all: a UDF, a non-deterministic function, or an expression over something the source cannot see is unpushable, so Catalyst leaves it above the scan. Second, the data source accepts only a subset of filters — Parquet handles simple comparisons on primitive columns, and an implicit cast introduced by comparing a string column to a date can stop the match. Third, a filter can be partially pushed: Spark pushes what it can and keeps the residual above the scan, so a Filter node does not by itself prove pushdown failed. Read PushedFilters and PartitionFilters together, then check Input Size / Records on the scan, because that number is the only proof the pruning reduced I/O. PartitionFilters eliminate whole directories; PushedFilters only skip row groups inside files using Parquet footer statistics.

df.filter("order_date >= date'2024-01-01'").explain("formatted")
# read PartitionFilters, PushedFilters, and Input Size on the scan

Senior depth: Because row-group skipping depends on physical layout. If the filtered column's values are spread evenly across every file, the min/max statistics in each footer overlap the predicate and no row group can be skipped. That is the argument for clustering or Z-ORDER on the predicate column: pushdown gives Spark permission to skip, layout is what makes skipping possible.

Common mistake: Concluding pushdown failed just because a Filter node is still present above the scan.

Follow-up they will ask: Why would a correct PushedFilter still not reduce bytes read?

Lesson · Simulation

senior

A nine-table join planned left-deep in exactly the order you wrote it, and the intermediate results are enormous. What do you check?

Catalyst optimizer · tap to open the answer

Say this first: Whether statistics exist at all — without them Catalyst cannot reorder joins and keeps your written order.

Remember these

  • Join reorder is CBO, gated by config and a relation-count threshold
  • No column statistics means no reordering, so your written order stands
  • explain("cost") shows the rowCount Catalyst is reasoning from

Why: Join reordering is a cost-based rule gated by spark.sql.cbo.enabled and spark.sql.cbo.joinReorder.enabled, and it only considers up to spark.sql.cbo.joinReorder.dp.threshold relations. All of that is useless without row counts and column statistics, so run ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS on the participating tables and re-plan. Then read explain("cost") and compare the estimated sizeInBytes and rowCount on each join node against reality. If the estimates are still wrong — common when a side is filtered through a UDF or read over JDBC — restructure the query so the most selective join runs first, or materialize the selective intermediate. Dropping a broadcast hint on every small side is the shortcut people reach for, and it works until one of those dimensions grows.

spark.conf.set("spark.sql.cbo.enabled", "true")
spark.conf.set("spark.sql.cbo.joinReorder.enabled", "true")
df.explain("cost")

Senior depth: When the selectivity is something statistics can never express — a predicate over a UDF, a semi-join against a set computed at runtime, or a source that reports no stats. Then you encode the knowledge in the query: filter and materialize the selective side first so the optimizer is handed a small relation instead of being asked to guess one. Document why, because the next engineer will see a hand-ordered join and try to tidy it up.

Common mistake: Assuming Spark always reorders joins because 'the optimizer handles it'.

Follow-up they will ask: When is rewriting the query better than fixing the statistics?

Lesson · Simulation

architect

The top of your explain output says AdaptiveSparkPlan isFinalPlan=false. What does that tell you about everything underneath it?

Catalyst optimizer · tap to open the answer

Say this first: That you are reading the pre-runtime guess, and the operators below may be replaced once real shuffle statistics arrive.

Remember these

  • isFinalPlan=false means a pre-execution, estimate-based plan
  • AQE re-plans only at Exchange boundaries, once stats arrive
  • The SQL / DataFrame tab shows the final plan that actually ran

Why: AdaptiveSparkPlan is the wrapper AQE puts at the root of a query. isFinalPlan=false means the query has not executed, so what you see is the physical plan Catalyst produced from estimates. Once a shuffle map stage completes, AQE reads its map output statistics and can coalesce partitions, convert a SortMergeJoin to a BroadcastHashJoin, or split skewed partitions, so the plan that actually ran may differ. To see the real plan, run the query and open the SQL / DataFrame tab, where the node reports isFinalPlan=true along with the changes AQE applied. The design consequence is that an explain() pasted into a design doc is a hypothesis, and a performance review should cite the final plan from the UI instead.

df.collect()
# then open the SQL / DataFrame tab: AdaptiveSparkPlan isFinalPlan=true

Senior depth: Capture it from the Spark event log or a query execution listener rather than from a notebook — the SQL tab is backed by events you can retain and query. For a platform, keep event logs, record the final plan per query, and alert when the join strategy changes; a flip from BroadcastHashJoin to SortMergeJoin is usually the earliest signal that a dimension outgrew its threshold.

Common mistake: Treating a notebook explain() as the plan that ran, then debugging operators AQE already replaced.

Follow-up they will ask: How do you capture the final plan for a job that runs on a schedule?

Lesson · Simulation

senior

WholeStageCodegen is off for a stage. How do you find why?

Tungsten execution engine · tap to open the answer

Say this first: An operator that can't be compiled — often a UDF, an unsupported expression, or a fallback.

Why: explain() and SQL UI: look for the break in the * codegen star. Fix the expression. Don't 'tune Tungsten' as a config cult.

Common mistake: Random spark.tungsten flags from a blog.

Follow-up they will ask: How does Photon relate to Tungsten on Databricks?

Lesson · Simulation

advanced

What does whole-stage codegen actually generate, and why does fusing operators help the CPU rather than only saving function calls?

Tungsten execution engine · tap to open the answer

Say this first: It generates one Java method with the operator chain inlined as a loop, so values stay in local variables instead of travelling through boxed rows.

Remember these

  • One generated method replaces a chain of iterator next() calls
  • Values stay in local variables instead of boxed row objects
  • Sequential reads over binary rows make CPU caches effective
  • explain("codegen") prints the generated Java source

Why: Without codegen each operator is an iterator that pulls a row from its child through a virtual call, which defeats branch prediction and forces the row to be materialized at every step. Codegen emits one method where filter conditions, projections, and aggregate updates are inlined into a loop over the input, so intermediate values live in locals the JIT can keep in registers. Combined with Tungsten's binary row format — fixed-width fields at known offsets in a contiguous buffer — the loop reads memory sequentially, which is what makes hardware prefetching and cache lines pay off. You can read the generated source with df.explain("codegen"). The limit is method size: HotSpot will not JIT-compile a method past roughly 64 KB of bytecode, so very long operator chains have to fall back.

df.filter("amount > 0").select("id").explain("codegen")

Senior depth: spark.sql.codegen.hugeMethodLimit caps the bytecode size Spark will emit for a codegen stage, because past the JVM's method size limit HotSpot refuses to JIT and the method would run interpreted — slower than the non-codegen path. When a stage exceeds it, Spark falls back to the iterator model for that stage. spark.sql.codegen.maxFields bounds how many schema fields a codegen stage will handle, which is why very wide tables lose their stars with no UDF involved.

Common mistake: Describing codegen as 'avoiding function call overhead' and stopping there, missing the memory-layout half.

Follow-up they will ask: What makes a generated method too big to compile?

Lesson · Simulation

advanced

A stage lost whole-stage codegen even though the plan contains no UDF. What causes would you check?

Tungsten execution engine · tap to open the answer

Say this first: Too many schema fields, a generated method past the size limit, a compile failure that fell back, or an operator with no codegen support.

Remember these

  • Wide schemas trip spark.sql.codegen.maxFields
  • Long chains trip spark.sql.codegen.hugeMethodLimit
  • A compile failure falls back silently and only shows in executor logs

Why: spark.sql.codegen.maxFields bounds the number of schema fields a codegen stage will handle, so a very wide table can disable it by width alone. spark.sql.codegen.hugeMethodLimit bounds generated bytecode size, and a long operator chain or a huge CASE expression can exceed it, at which point that stage runs through the iterator path. The code generator can also fail to compile the generated source, and with fallback enabled Spark logs the failure and runs interpreted instead of failing the query — so check executor logs for the compile error rather than assuming your plan is at fault. Finally, some operators have no codegen implementation and are always unstarred. The fix is almost always to narrow the projection or shorten the chain, not to flip Tungsten configs.

spark.conf.get("spark.sql.codegen.maxFields")
spark.conf.get("spark.sql.codegen.hugeMethodLimit")

Senior depth: Read executor stderr for the codegen compile exception and the generated source dump — Spark logs the failing class. Reproduce with a narrowed projection to find which expression blows the method up, then split the query at that point so each half compiles. Setting spark.sql.codegen.wholeStage to false globally is a legitimate experiment and a terrible fix.

Common mistake: Assuming any missing star means a UDF, and hunting for one that is not there.

Follow-up they will ask: How would you confirm a compile failure rather than guess at it?

Lesson · Simulation

senior

Make the memory and garbage-collection argument for UnsafeRow, not the CPU one.

Tungsten execution engine · tap to open the answer

Say this first: A binary row is one contiguous buffer with no per-field object header, so it uses far less space and the collector has almost nothing to trace.

Remember these

  • One buffer per row, with no object header per field
  • The collector traces one object instead of one per field
  • Off-heap execution memory removes that memory from GC entirely
  • Off-heap still counts against the container's total limit

Why: A row modelled as Java objects costs an object header per field plus pointers, and a String carries a character array of its own, so the in-memory footprint can be several times the logical data size. UnsafeRow lays fields out at fixed offsets in a single byte buffer — null bits, then fixed-width values, then a variable-length region — so one row is one allocation. That also means the garbage collector traces one object instead of thousands, which is what keeps GC time flat when a task holds millions of rows in a hash aggregation. Turning on spark.memory.offHeap.enabled with spark.memory.offHeap.size moves those buffers out of the heap entirely, so execution memory is managed by Spark rather than by the collector. The caveat is that off-heap memory still counts against the container's total limit, so it trades GC pressure for careful overhead budgeting.

spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "4g")

Senior depth: They are converted to Arrow record batches, shipped to a Python worker, and rebuilt as pandas objects, so you pay the serialization and hold both representations at once. That memory lives in the Python process, outside the JVM heap, which is why those jobs are killed by the cluster manager while Spark's heap metrics look healthy. Budget spark.executor.memoryOverhead for it and keep the Arrow batch size modest.

Common mistake: Treating Tungsten as purely a CPU story, then being surprised a Python UDF raises memory as well as latency.

Follow-up they will ask: What happens to those rows when a pandas UDF runs?

Lesson · Simulation

architect

Where does the vectorized Parquet reader sit relative to Tungsten, and when does it stop being used?

Tungsten execution engine · tap to open the answer

Say this first: It fills Tungsten's columnar batches straight from Parquet pages, and it is bypassed for schemas it cannot handle.

Remember these

  • The vectorized reader produces ColumnarBatch that codegen'd scans read directly
  • spark.sql.parquet.enableVectorizedReader gates it
  • Unsupported schemas fall back to a row-based read
  • Flat schemas keep the whole read path in binary form

Why: The vectorized reader decodes Parquet column chunks into ColumnarBatch objects — a few thousand rows at a time in columnar buffers — instead of constructing one row at a time, and codegen'd scans consume those batches directly. That keeps the whole read path in the binary world: no Java object per value, dictionary encoding preserved where possible, and sequential memory access. It is gated by spark.sql.parquet.enableVectorizedReader and bounded by spark.sql.parquet.columnarReaderBatchSize, and it is skipped for schemas it does not support, historically deeply nested structures, in which case the plan falls back to a row-based read. Architecturally that is an argument for flat, typed schemas at the silver layer: nested JSON costs you the vectorized path before any of your query logic runs.

spark.conf.get("spark.sql.parquet.enableVectorizedReader")
spark.conf.get("spark.sql.parquet.columnarReaderBatchSize")

Senior depth: Check whether the scan node reports columnar output, and compare scan time per byte against a flattened copy of the same data. The practical test is to project only the flat columns: if the scan speeds up far more than the byte reduction explains, the nested column was costing you the vectorized path. Then fix it at ingestion by flattening in bronze rather than per query.

Common mistake: Blaming Parquet for a slow scan when the schema pushed the read off the vectorized path.

Follow-up they will ask: How would you prove a table lost the vectorized reader?

Lesson · Simulation

senior

One task spilled 60 GB to disk and finished. A different task's container was killed. Same memory problem?

Executor memory: execution wins, cache loses · tap to open the answer

Say this first: No — spill is Spark managing execution memory; a killed container is the cluster manager seeing total process memory exceed its limit.

Why: Spark tracks execution memory and spills sorted runs to local disk when it runs short, so the task survives slowly and you see Spill (Memory) and Spill (Disk) on the stage. YARN or Kubernetes knows nothing about that accounting — it sees the whole process: JVM heap plus off-heap buffers plus Python workers. Cross the container limit and it is killed with no Spark-level OutOfMemoryError anywhere.

Senior depth: spark.executor.memoryOverhead — the non-heap headroom for Python workers, off-heap buffers, and shuffle. On PySpark and pandas UDF workloads that is usually what is actually being exceeded.

Common mistake: Raising spark.executor.memory to fix a container kill, which just asks for a limit it still overshoots.

Follow-up they will ask: Which setting do people forget in that second case?

Lesson · Simulation

architect

Why does Tungsten's binary row format belong in a memory conversation, not just a CPU one?

Executor memory: execution wins, cache loses · tap to open the answer

Say this first: Binary rows outside the Java object model use far less space per row and stop the GC from walking your data.

Why: Tungsten keeps rows in compact buffers with explicit offsets instead of JVM objects, so the same records occupy much less memory and the collector ignores them. Turning on spark.memory.offHeap.enabled with a size moves execution memory off the heap entirely, which keeps GC pauses down — but that memory still counts against the container's total limit.

Senior depth: Python UDFs — rows leave the binary format and become Python objects, so you pay the conversion and the memory back. That is a memory argument for built-ins, not only a CPU one.

Common mistake: Enabling off-heap memory without raising the container's overall allowance.

Follow-up they will ask: What destroys the Tungsten advantage in a PySpark pipeline?

Lesson · Simulation

advanced

Walk me from 'execution memory is full' to an actual OutOfMemoryError. What are the steps in between?

Executor memory: execution wins, cache loses · tap to open the answer

Say this first: Borrow free space, evict cached blocks down to the storage floor, spill sorted runs to disk, and only then fail.

Remember these

  • Order: borrow free space → evict cache → spill to disk → fail
  • Eviction is one-directional; storage can never reclaim from execution
  • OOM means something unspillable did not fit, not that spill failed
  • A container kill is the process limit, not the heap

Why: A task asks the memory manager for execution memory. If the unified pool has free space it is granted; if storage is using space above spark.memory.storageFraction, those cached blocks are evicted to make room, because eviction only runs in that direction. When nothing more can be reclaimed, the sorter or the aggregation map serializes its contents to a local disk file as a sorted run and continues with an empty buffer, repeating as needed and merging the runs at the end — that is the Spill (Memory) and Spill (Disk) pair. The JVM only throws OutOfMemoryError when something unspillable does not fit: a single record or key group larger than the available page, an untracked user-code allocation, or a heap exhausted by blocks persisted MEMORY_ONLY. A container kill is a different failure entirely — the process exceeding its total limit, heap plus off-heap plus Python — and Spark's accounting never sees it.

# stage page: Spill (Memory), Spill (Disk), Peak Execution Memory
# executor log: OutOfMemoryError vs a container-killed message

Senior depth: Eviction. There is no counter for blocks evicted by execution on the stage page — you infer it from Fraction Cached on the Storage tab falling between queries, and from a later plan reverting to FileScan where it previously showed InMemoryTableScan. That is the mechanism behind a notebook that gets slower over a session with no error anywhere.

Common mistake: Explaining it as 'memory fills and Spark OOMs', skipping eviction and spill entirely.

Follow-up they will ask: Which of those steps is invisible in the Spark UI?

Lesson · Simulation

senior

A stage spends 40% of task time in GC, with no spill and no OOM. What do you do?

Executor memory: execution wins, cache loses · tap to open the answer

Say this first: Find what is holding long-lived objects — usually cache or user-code structures — before touching heap size.

Remember these

  • GC time with no spill is a live-set problem, not an execution-memory one
  • A bigger heap can lengthen pauses instead of shortening them
  • Executors tab: compare GC Time against Task Time, per executor

Why: High GC with no spill means the collector is working hard, not that execution memory ran short, so raising spark.executor.memory often makes it worse by lengthening full-collection pauses. Check the Storage tab first: a large MEMORY_ONLY cache keeps millions of live objects that every collection has to trace. Then look at user memory — a per-partition dictionary, a growing accumulator, or Python worker churn on the same host. Enabling off-heap execution memory with spark.memory.offHeap.enabled moves Tungsten's buffers out of the collector's reach, and smaller executors with fewer cores shrink the live set each JVM manages. In the Executors tab, compare GC Time against Task Time per executor: if one host dominates, suspect cache placement or a noisy neighbour rather than the job itself.

df.unpersist()
spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "4g")

Senior depth: When the live set per JVM is the problem rather than total capacity. Several smaller executors give several smaller heaps, each with shorter pauses and an independent collector, at the cost of more broadcast copies and more shuffle connections. The trade is real: fewer, larger executors reduce broadcast duplication and shuffle fan-out, while more, smaller executors reduce GC pause and the blast radius when one dies.

Common mistake: Treating GC time as a signal to request a larger instance type.

Follow-up they will ask: When is a smaller executor the better answer?

Lesson · Simulation

architect

Same budget, two shapes: eight executors with sixteen cores, or thirty-two with four. Which for a shuffle-heavy job?

Executor memory: execution wins, cache loses · tap to open the answer

Say this first: Mid-size executors, because execution memory is divided by concurrent tasks and both extremes hurt in different ways.

Remember these

  • Execution memory per task ≈ unified pool ÷ cores on the executor
  • Huge executors: long GC pauses and a bigger blast radius
  • Tiny executors: a broadcast copy each and more shuffle connections
  • Validate with Peak Execution Memory per task, not with a rule

Why: Every core on an executor runs a task concurrently and all of them draw from the same unified pool, so a sixteen-core executor gives each task roughly a sixteenth of the execution memory — which is how a large executor still spills. Very large executors also mean long GC pauses, a bigger loss when one dies, and heavy object-store contention from many threads in one JVM. Very small executors waste memory on per-executor overhead, hold a broadcast copy each, and multiply shuffle connections, which makes fetch failures more likely. The usual landing zone is four to five cores per executor with spark.executor.memoryOverhead budgeted on top, then sizing the count for total parallelism. Validate with Peak Execution Memory and Spill (Disk) per task rather than with a rule of thumb.

--executor-cores 4 --executor-memory 16g --conf spark.executor.memoryOverhead=3g

Senior depth: Python workers live outside the heap, roughly one per task slot, so a high-core executor multiplies Python processes against a single memoryOverhead budget. For pandas UDF workloads keep cores per executor lower and overhead higher, and cap the Arrow batch size — otherwise you converge on container kills with a heap chart that looked healthy the whole way down.

Common mistake: Maximising cores per executor to 'use the machine fully', then raising memory to stop the resulting spill.

Follow-up they will ask: What changes this answer for a PySpark-heavy job?

Lesson · Simulation

senior

A stateful aggregation ran clean for a week, then started OOMing every few hours. First hypothesis?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: No watermark, so state retained every key forever and finally outgrew the executors.

Why: Without withWatermark the engine cannot know a group will never receive late data, so its state is kept indefinitely. The streaming progress metrics show the state operator's numRowsTotal climbing monotonically and state store commit time growing with it. A watermark lets keys older than the event-time threshold be dropped.

Senior depth: Events later than the watermark are dropped and cannot be retrofitted — you are signing up to a bounded lateness contract, so route stragglers to a repair batch job if the business needs them.

Common mistake: Restarting the stream on a bigger cluster, which just moves the failure a few days out.

Follow-up they will ask: What do you give up when you add the watermark?

Lesson · Simulation

architect

Your foreachBatch writes to a transactional store and a restart happens mid-batch. How do you keep the sink correct?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Make the write idempotent and keyed, because the same batchId can be re-executed.

Why: foreachBatch hands you the DataFrame and the batchId, and on failure the engine may replay a batch that already wrote part of its rows. Use business keys (or the batchId) in a MERGE / upsert, or record the last committed batchId in the sink transactionally so a replay becomes a no-op. Output mode matters too: append emits only new rows, update emits changed keys, complete rewrites the whole result and only works for small aggregates.

Senior depth: When latency tolerance is hours: AvailableNow consumes everything currently available across several micro-batches and stops, so you keep streaming semantics and checkpointed offsets without paying for a cluster that never sleeps.

Common mistake: Assuming the checkpoint's exactly-once guarantee extends to an arbitrary external sink you wrote yourself.

Follow-up they will ask: When would you pick Trigger.AvailableNow over an always-on stream?

Lesson · Simulation

advanced

A batch failed halfway and the stream restarted. Some rows had already reached the Delta sink. Why are there no duplicates?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: The offset log is written before a batch runs and the commit log after, and the sink records which batch it last wrote.

Remember these

  • Offsets are written before the batch, commits after — that gap drives replay
  • Replay reuses the identical offset range, so the batch is deterministic
  • The sink must dedupe on batch id or business key for this to hold

Why: Before executing batch N the engine writes its offset range to the write-ahead offset log in the checkpoint directory. Only when the batch completes does it append to the commit log. On restart the engine finds an offset entry with no matching commit, so it replays batch N over exactly the same offset range — determinism of that range is half the guarantee. The other half is the sink: the Delta sink records the stream's identifier and the last batch id it committed in the table's transaction log, so a replay of an already-written batch is recognised and skipped. That combination is what people mean by end-to-end exactly-once, and it is why the guarantee does not extend automatically to a sink you wrote yourself inside foreachBatch.

(df.writeStream
   .option("checkpointLocation", "/chk/orders")
   .toTable("gold.orders"))

Senior depth: Then you have at-most-once at best. A source that cannot be re-read deterministically — a queue that acknowledges on read, or a file source whose files a retention job deleted before the replay — leaves the engine unable to reconstruct the batch. That is why Kafka retention has to exceed your worst-case outage window, and why file sources need a cleanup policy that respects the stream's committed position.

Common mistake: Attributing exactly-once to the checkpoint alone, with no requirement placed on the sink.

Follow-up they will ask: What breaks this if the source cannot replay the same offsets?

Lesson · Simulation

advanced

Both sides of a stream-stream join have watermarks, and state still grows without bound. What is missing?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: A time-range condition in the join predicate — watermarks alone do not tell Spark when a buffered row can be dropped.

Remember these

  • Watermarks on both sides are necessary but not sufficient
  • The join needs a time-range predicate or an event-time window equality
  • numRowsTotal that never plateaus is the signature

Why: For a stream-stream join the engine buffers rows from both sides because a match may arrive later. To evict a buffered row it must prove no future row can match it, and that needs two things: a watermark on each side's event-time column, and a constraint relating the two event times, either a range condition in the predicate or a join on an event-time window. With watermarks alone the predicate is unbounded in time, so nothing can be proven stale and state keeps every row forever. Add a bounded condition between the two timestamps and the engine can compute an eviction threshold. Watch stateOperators numRowsTotal in the progress report — if it never plateaus after the watermark has advanced, the missing time condition is why.

orders.join(clicks,
  (orders.id == clicks.order_id) &
  (clicks.ts.between(orders.ts, orders.ts + expr("INTERVAL 30 MINUTES"))))

Senior depth: An outer join cannot emit an unmatched row until it is certain no match will arrive, so it holds the row until the watermark passes the range bound and only then emits it with nulls. Outer results are therefore delayed by the full range plus the watermark, and state is strictly larger than an inner join's. Make sure downstream consumers expect that delay before choosing an outer join in a streaming pipeline.

Common mistake: Assuming withWatermark on both inputs is enough for state eviction in a stream-stream join.

Follow-up they will ask: What does an outer stream-stream join add on top of that?

Lesson · Simulation

senior

Batch duration looks fine but the stream is steadily falling behind the source. Where do you look, and what do you change?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Compare processed rows per second to input rows per second, then bound batch size and attack the per-batch overhead.

Remember these

  • inputRowsPerSecond above processedRowsPerSecond is the backlog signal
  • durationMs separates real work from per-batch overhead
  • Kafka partition count caps source-read parallelism
  • 200 shuffle partitions is usually wrong for a small stateful batch

Why: In the query progress, inputRowsPerSecond above processedRowsPerSecond means the backlog is growing however healthy each batch looks. Break the batch down with durationMs: time in addBatch is real work, while large latestOffset or walCommit values mean per-batch overhead dominates and you are running too many small batches. For Kafka, maxOffsetsPerTrigger bounds rows per batch so a backlog drains predictably instead of in one enormous batch; for files, maxFilesPerTrigger or maxBytesPerTrigger does the same. Then check parallelism: a topic with eight partitions caps source read parallelism at eight tasks no matter the cluster size, and spark.sql.shuffle.partitions left at 200 for a small stateful batch spends the batch on task overhead. Fix the source partition count and the shuffle partition count together.

(spark.readStream.format("kafka")
   .option("maxOffsetsPerTrigger", 500000)
   .option("subscribe", "orders").load())

Senior depth: Because state is partitioned by that number and stored under the checkpoint, so a stateful query fixes it when the checkpoint is created and refuses to start if you change it. Choose it deliberately before the first run — low enough that batches are not dominated by task overhead, high enough to carry future state volume — because changing it later means a state migration or a fresh checkpoint.

Common mistake: Adding executors when the source has fewer partitions than the cluster already has task slots.

Follow-up they will ask: Why is spark.sql.shuffle.partitions awkward to change for a stateful query?

Lesson · Simulation

architect

You need to add a column to a streaming aggregation's grouping key, and the query will not start against the existing checkpoint. What is your migration plan?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Treat it as a new query with a new checkpoint, resumed at the old query's committed offsets, with the history backfilled in batch.

Remember these

  • Changing the grouping key changes state schema; resume is refused by design
  • Start the new query at the old one's committed offsets, not at latest
  • Backfill history in batch and reconcile before cutting consumers over
  • Version checkpoint paths so a schema change is a deploy, not an outage

Why: The state store is keyed by the grouping key, so changing the key changes the state schema and the engine refuses to resume rather than produce wrong results. There is no in-place migration for that state, so the plan is: stop the old query cleanly, record its last committed offsets from the commit log, start the new query against a new checkpoint directory at exactly those offsets so there is no gap and no overlap, and reprocess the historical window in a batch job that writes the new grain. Run both outputs into separate tables and reconcile counts before cutting consumers over. Build this into the design: one checkpoint per query, checkpoint paths carrying a version suffix, and sinks written through an idempotent MERGE so a replay or an overlap is recoverable rather than a data incident.

# new checkpoint path, source resumed at the old query's committed offsets
(spark.readStream.format("kafka")
   .option("startingOffsets", last_committed_json).load())
# writeStream.option("checkpointLocation", "/chk/orders_v2")

Senior depth: Whenever there is any gap between stopping the old query and starting the new one, because latest skips everything produced in between and nothing reports it — counts are simply lower. Resume from the recorded committed offsets instead, and if you must use latest, pair it with a batch backfill bounded by the two timestamps and reconcile row counts before anyone trusts the table.

Common mistake: Deleting the old checkpoint so the new code starts, then finding the gap or the duplicates in the gold table a day later.

Follow-up they will ask: When does startingOffsets 'latest' quietly lose data?

Lesson · Simulation

senior

How do you prove skew versus a slow node?

Data skew · tap to open the answer

Say this first: Skew: same task id always huge shuffle read for a key. Slow node: different keys, one host.

Why: Look at shuffle read bytes, not just duration. If bytes are even but duration isn't, suspect disk/CPU/noisy neighbor. If bytes are skewed, it's data.

Senior depth: Add salt, agg/join on (key, salt), then drop salt and re-agg if needed.

Common mistake: Speculative execution as the root-cause fix.

Follow-up they will ask: Walk through salting without ruining the grain.

Lesson · Simulation

advanced

Walk me through salting a join, not an aggregation. What do you do to each side?

Data skew · tap to open the answer

Say this first: Add a random salt to the skewed side's key and replicate the other side across every salt value, then join on the composite key.

Remember these

  • Skewed side gets a random salt; the other side is replicated across all salts
  • Join on (key, salt), then drop the salt from the output
  • Salting every key multiplies the other side's shuffle by N
  • Scope the salt to the known hot keys to contain the cost

Why: Salting a join is asymmetric. On the skewed side you append a random integer in a fixed range to the join key, which splits the hot key's rows across that many partitions. On the other side you cannot randomise, because a row must still meet every bucket of its key, so you explode it: cross each row with the full range of salt values so one row becomes N. Then join on key plus salt and drop the salt column from the output. The result is correct because every (key, salt) pair on the skewed side has a matching row on the exploded side. The cost is that explosion — the other side is N times larger through the shuffle — so scope the salt to the known hot keys rather than applying it to every key, typically with a conditional salt that is zero for everything else.

N = 16
f = fact.withColumn("salt", (F.rand() * N).cast("int"))
d = dim.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(N)])))
f.join(d, ["key", "salt"]).drop("salt")

Senior depth: From the ratio you need to fix, not from a round number. Take the hot partition's Shuffle Read and divide by the median partition size — that quotient is roughly the N that flattens it. A larger N splits the hot key further but multiplies the replicated side, so there is a point where the explosion costs more than the straggler. Measure max against median Shuffle Read after the change rather than assuming it worked.

Common mistake: Salting only the fact side and losing every row whose key hashed to a salt the dimension does not have.

Follow-up they will ask: How do you choose N?

Lesson · Simulation

advanced

The same hot key makes a join unusable but a groupBy().count() on it finishes fine. Why the difference?

Data skew · tap to open the answer

Say this first: Aggregation does a partial pass on the map side, so the hot key reaches the reducer already collapsed; a join has to move every row.

Remember these

  • The partial HashAggregate collapses the hot key before the shuffle
  • A join must move every row of the hot key across the Exchange
  • collect_list, collect_set, and distinct defeat the partial pass

Why: A reducing aggregation plans as two HashAggregate nodes around the Exchange: the partial one combines rows per key within each input partition, so a hot key contributes one partial row per partition instead of millions of raw rows. By the time the shuffle happens there is very little data for that key, and the reduce-side HashAggregate finishes quickly. A join cannot do that, because matching requires the actual rows, so every row of the hot key crosses the shuffle into one partition and one task hashes or sorts all of it. The exception is an aggregation that cannot combine incrementally: collect_list, collect_set, and count(DISTINCT ...) without a rewrite keep per-key data proportional to the key's row count, so they skew exactly like a join. Read the plan — two HashAggregate nodes means the map-side combine is happening, one means it is not.

df.groupBy("key").count().explain("formatted")
# two HashAggregate nodes around the Exchange = map-side combine

Senior depth: Stop collecting the full list. Either aggregate to the statistic you actually need so the partial pass can combine it, cap the list with a windowed top-N before collecting, or branch the hot key into its own job. If the full list is genuinely required, two-stage it: collect per (key, salt), then merge the lists per key, which keeps any single task's memory bounded.

Common mistake: Treating skew as one phenomenon and expecting identical symptoms from aggregations and joins.

Follow-up they will ask: How would you make a skewed collect_list survivable?

Lesson · Simulation

senior

Every shuffle stage is even, but the final write has one task running for fifty minutes. What is going on?

Data skew · tap to open the answer

Say this first: A partitioned write funnels all rows for one output partition value into one task, so a skewed partition column becomes a skewed writer.

Remember these

  • partitionBy funnels one output value into one writing task
  • Signature: even shuffle stages, one long task in the write stage
  • maxRecordsPerFile bounds the output a single task writes per file
  • A skewed partition column is a layout mistake, not a tuning problem

Why: With partitionBy on the write, Spark wants each output directory written by as few tasks as possible, so plans typically include a shuffle or a sort on the partition columns; one hot value then lands in one task that writes the entire directory. Dynamic partition overwrite and sorted writes amplify it, because the task may also hold or sort rows per open file. The signature is clean shuffle stages followed by a write stage whose task duration histogram has a single long bar, high Spill (Disk) on that task, and one output file far larger than the rest. The cheapest fix is to repartition on the partition columns plus a bounded salt before the write so the hot value is split across a known number of files, or to set maxRecordsPerFile so one task cuts its output into bounded files. If the partition column is badly skewed or high cardinality, the real answer is to stop partitioning by it.

N = 8
(df.withColumn("salt", (F.rand() * N).cast("int"))
   .repartition("event_date", "salt").drop("salt")
   .write.partitionBy("event_date").save(path))

Senior depth: Splitting a hot value across N writers produces N files for that partition instead of one, so you trade a straggler for small-file pressure. Size N from the target file size — divide the partition's bytes by roughly 128 MB to 1 GB depending on the reader — and compact on a schedule. Liquid clustering and OPTIMIZE exist precisely so you do not have to choose between a straggler and a thousand tiny files forever.

Common mistake: Looking only at shuffle stages for skew and ignoring the write stage's task histogram.

Follow-up they will ask: What does this do to the number of output files?

Lesson · Simulation

architect

The hot key is 40% of volume, the business will not change the key, and salting is already in place. Design the pipeline.

Data skew · tap to open the answer

Say this first: Split the pipeline by key class: a dedicated path for the hot keys and the ordinary path for the long tail.

Remember these

  • At 40% of volume it is two workloads, not one unbalanced shuffle
  • The hot path gets its own parallelism, layout, and incremental strategy
  • Classify hot keys from profiling data, not from a hard-coded list
  • Add a reconciliation step so the two paths provably sum to the whole

Why: Once a single key is a large share of volume you are not balancing a shuffle any more, you are running two workloads in one job. Isolate the hot keys into their own branch with their own parallelism, their own output layout, and where the computation allows a pre-aggregation or incremental model, so the full history is never reshuffled. Run the long tail through the ordinary path where the hash partitioner behaves. The hot path can also change shape: a key that is 40% of a fact usually supports incremental maintenance — process only new rows and merge into a stored aggregate — instead of a daily full recompute. Encode the key classification as data rather than a hard-coded list, so a new hot key is detected by a profiling job and routed automatically. The cost is two code paths to maintain and a reconciliation step to prove they add up.

Senior depth: Profile the join and groupBy keys as a scheduled job: rows per key, and the max-to-median ratio of the resulting partition sizes. Alert when a key crosses the share your current salt factor can absorb. That turns skew from an on-call page into a backlog item, and it also gives the hot-path routing table its input.

Common mistake: Increasing the salt factor every quarter instead of admitting the pipeline needs two paths.

Follow-up they will ask: How do you detect the next hot key before it causes an incident?

Lesson · Simulation

senior

Broadcast join killed the driver, not the executors. Walk the path.

Driver OOM · tap to open the answer

Say this first: Driver collects the build side to ship the broadcast.

Why: If the dimension is 6 GB, the driver must hold it. Executors OOM later if they also can't hold the hashed relation — but the first failure can be the driver. Measure broadcast exchange size.

Senior depth: Error class → driver vs executor → action/broadcast in the plan → rewrite or size the JVM with a documented cap.

Common mistake: Only looking at executor logs.

Follow-up they will ask: What's your evidence-ordered debug?

Lesson · Simulation

advanced

A job with no collect() anywhere still OOMs the driver after running four hundred thousand tasks. What else lives in driver heap?

Driver OOM · tap to open the answer

Say this first: Scheduler and UI bookkeeping — task metrics, accumulator updates, and retained job, stage, and SQL execution data for every task that ran.

Remember these

  • The driver holds per-task metrics and accumulator updates for every task
  • spark.ui.retained* and spark.sql.ui.retainedExecutions bound UI history
  • Dropped-event warnings in the driver log precede the memory problem
  • Fewer, larger partitions means less driver bookkeeping

Why: The driver keeps per-task metrics and accumulator updates for the UI and the event log, and it retains completed jobs, stages, and SQL executions up to spark.ui.retainedJobs, spark.ui.retainedStages, and spark.sql.ui.retainedExecutions. A job with hundreds of thousands of tasks, or a long-lived session running thousands of queries, accumulates that bookkeeping until the heap is gone, with no collect in sight. The listener bus is the related failure: events queue up to spark.scheduler.listenerbus.eventqueue.capacity, and if a listener is slow you see dropped-event warnings in the driver log before memory becomes the problem. The fixes are to cut the task count with fewer, larger partitions, trim the retained counts, and recycle long-lived sessions between large workloads. Chart the driver heap trend over the run rather than looking at any single stage.

spark.conf.get("spark.ui.retainedStages")
spark.conf.get("spark.sql.ui.retainedExecutions")

Senior depth: A session serving many queries never resets its retained history, so UI state, broadcast references, and accumulator data accumulate across all of them. On a shared cluster that turns one team's thousand-query notebook into everyone's driver OOM. Bound the retained counts, unpersist broadcast variables when finished with them, and recycle long-lived sessions on a schedule rather than treating uptime as a virtue.

Common mistake: Assuming driver OOM always means someone called collect(), and grepping the code instead of reading the heap trend.

Follow-up they will ask: How would a long-lived shared session make this worse?

Lesson · Simulation

senior

The driver dies before a single task is launched, on a table with millions of small files. What is it doing?

Driver OOM · tap to open the answer

Say this first: Listing and indexing files — the driver builds the scan's file index in its own heap before any stage exists.

Remember these

  • The driver builds and holds the file index before any stage is submitted
  • No stages in the UI plus a busy driver is the listing signature
  • parallelPartitionDiscovery speeds listing but not the index size
  • Compaction and sane partition columns are the durable fix

Why: Planning a file-based scan requires enumerating files and their sizes so Spark can build splits, and that index lives in driver memory. For a table with millions of files, or a deeply partitioned directory tree, that structure plus per-file metadata can exhaust the heap before the job starts — the Jobs tab shows nothing because no stage has been submitted. Spark parallelises the listing itself across executors once the directory count passes spark.sql.sources.parallelPartitionDiscovery.threshold, which speeds listing up but does not shrink the index the driver ends up holding. The signals are a long gap between submitting the query and the first stage, driver CPU busy with no tasks running, and listing activity in the driver log. The durable fix is file layout: compact small files, avoid partitioning on high-cardinality columns, and use a table format whose metadata supplies file lists without a full directory walk.

spark.conf.get("spark.sql.sources.parallelPartitionDiscovery.threshold")
# OPTIMIZE the table to cut file count before tuning anything else

Senior depth: Partitioning by a high-cardinality column — an id, an hour, a customer — creates a directory per value and often one small file per directory per write, so a table that is small in bytes can reach millions of files within weeks. That is a driver problem before it is a scan problem. Choose partition columns from query predicates and target partition size, aim for partitions in the hundreds of megabytes to gigabytes, and use clustering rather than partitioning for high-cardinality filters.

Common mistake: Reading the pre-stage gap as 'the cluster is still starting up' rather than as driver-side planning.

Follow-up they will ask: How does over-partitioning create this on a brand new table?

Lesson · Simulation

architect

How would you stop driver OOMs from reaching production across sixty jobs?

Driver OOM · tap to open the answer

Say this first: Make unbounded driver-bound actions a review failure, and give people a sanctioned way to get data out.

Remember these

  • Lint for collect, toPandas, and unbounded take with an allowlist
  • Ship sanctioned alternatives so the safe path is the convenient one
  • Keep maxResultSize bounded so failures stay legible
  • Monitor table file counts — listing is the other driver OOM

Why: Treat collect(), toPandas(), and unbounded take() in production code as lint failures with an explicit allowlist that requires a justification, because they are a small enumerable set of calls. Provide the sanctioned alternatives in a shared library — write to a table, export a bounded sample, compute approximate quantiles — so the easy path is also the safe one. Keep spark.driver.maxResultSize at a sane value rather than unlimited, so a mistake fails fast with a legible message instead of dying as a heap OOM at three in the morning. Standardise driver sizing per job class so a large driver is a reviewed choice rather than the default remedy. And monitor file counts per table, because the other driver OOM is listing, not collecting.

Senior depth: Three steps in order: confirm it is the driver by checking the Executors tab for completed tasks and no ExecutorLostFailure, identify the action or the planning phase from the driver log and the Jobs tab, and only then decide between rewriting the action and sizing the driver with a documented cap. Resizing first is how a data-path bug becomes a permanent line item.

Common mistake: Standardising on a large driver instance type as the policy, which pays for the bug forever.

Follow-up they will ask: What belongs in the runbook when it happens anyway?

Lesson · Simulation

senior

How do you choose between salting, more partitions, and more executor memory?

Executor OOM · tap to open the answer

Say this first: If one key is fat: salt or split. If all tasks are fat: more partitions or a smaller row (drop columns). Memory last.

Why: UI: shuffle read per task. One bar huge → skew. All bars huge → partition sizing or explode. GC thrash with even bytes → memory/config or too much cache.

Senior depth: Copy-pasting spark.memory.fraction from OSS blogs onto Photon/DBR without measuring.

Common mistake: A 2× memory ticket with no histogram attached.

Follow-up they will ask: What Spark config is a footgun with off-heap and Databricks?

Lesson · Simulation

advanced

An executor has 8 GB heap and 5 cores, and each task needs about 2 GB for its hash aggregation. Does it fit?

Executor OOM · tap to open the answer

Say this first: No — five concurrent tasks share one unified pool of roughly 4.5 GB, so each gets under a gigabyte and they spill or die.

Remember these

  • Unified pool ≈ (heap − reserve) × spark.memory.fraction
  • All cores on an executor draw from the same pool, concurrently
  • Peak Execution Memory per task is the demand side of the arithmetic
  • Fewer cores or smaller partitions beats a bigger heap divided by five

Why: Start from the heap: a fixed reserve comes off the top, then spark.memory.fraction — 0.6 by default — of the remainder forms the unified execution and storage pool, so 8 GB yields something like 4.5 GB of managed memory. Every core runs a task concurrently and all of them draw from that one pool, so with five cores the fair share is under a gigabyte per task, not 2 GB. The tasks will borrow free storage space, then spill sorted runs to disk, and only fail outright if a single unspillable record group does not fit. Peak Execution Memory per task on the stage page tells you what the tasks actually demanded. The two honest fixes are fewer cores per executor, so each task gets a larger share, or smaller partitions, so each task needs less — raising heap alone still divides by five.

# stage page: Peak Execution Memory, Spill (Memory), Spill (Disk)
# then try --executor-cores 3 before --executor-memory 16g

Senior depth: Because spill and GC are not free. With fewer concurrent tasks each one gets enough execution memory to finish in memory, so you remove the spill write, the spill read, and the merge from the critical path, and the JVM's live set shrinks so pauses get shorter. You lose parallelism per executor and usually get it back by running more executors. Prove it with Spill (Disk) going to zero and stage time dropping, not with the config diff.

Common mistake: Comparing a task's memory need against the full executor heap instead of against the per-task share.

Follow-up they will ask: Why might lowering executor cores make the job faster as well as safer?

Lesson · Simulation

advanced

The executor OOM happens in the write stage, not in any shuffle. What causes that?

Executor OOM · tap to open the answer

Say this first: A partitioned write holds one open writer per output partition value, each with its own buffer, so high cardinality in one task exhausts the heap.

Remember these

  • One open writer per output partition value, each with a row-group buffer
  • Unsorted input means many writers held open in one task at once
  • Signature: OOM in the final stage with no shuffle skew
  • repartition on the partition columns before the write

Why: Writing with partitionBy means a task may hold rows for many partition values, and each open output file carries a Parquet writer with its own in-memory row group buffer — tens or hundreds of megabytes each when row groups are large. If the data reaching a task is not sorted by the partition columns, it can open a writer for every value it encounters and hold them all simultaneously. Spark mitigates this by sorting on the partition columns before the write, but a plan that skips the sort, or a task that genuinely carries many values, still blows up. The signature is an OOM in the final stage with no Shuffle Read skew, plus an output directory with many small files written by the same task. The fixes are to repartition on the partition columns so each task writes few values, to cap rows per file with maxRecordsPerFile, or to reduce the Parquet row group size so each open writer buffers less.

(df.repartition("event_date", "country")
   .write.partitionBy("event_date", "country")
   .option("maxRecordsPerFile", 1000000).save(path))

Senior depth: Dynamic partition overwrite stages output and then swaps partitions, so the task still holds the same open writers and buffers, plus the commit protocol's bookkeeping. It makes the write heavier, not lighter. If a job both overwrites dynamically and carries many partition values per task, repartition on the partition columns first and keep the value count per task small — that one change addresses both failure modes.

Common mistake: Looking only at shuffle stages when the OOM is in the write, and concluding the data must be skewed.

Follow-up they will ask: How does this interact with dynamic partition overwrite?

Lesson · Simulation

senior

One executor dies and the job takes four times longer, not one twenty-fourth longer. Why does losing one executor cost that much?

Executor OOM · tap to open the answer

Say this first: Its shuffle files die with it, so downstream fetches fail and the DAGScheduler resubmits whole map stages.

Remember these

  • Losing an executor loses its shuffle output, not just its compute
  • FetchFailedException resubmits a parent stage, not just a task
  • A memory-caused death often repeats on the replacement executor
  • External shuffle service or shuffle tracking bounds the blast radius

Why: Shuffle output is written to the dead executor's local disk, and without an external shuffle service those blocks are gone. Reduce tasks fetching them raise FetchFailedException, the DAGScheduler unregisters the map output and resubmits the missing partitions of the parent stage, and everything that depended on them is redone. If the cause was memory pressure, the replacement executor often inherits the same fat partition and dies too, so you get a retry loop that burns the stage retry budget before failing the job. Cached blocks on that executor are lost and recomputed as well, and the retried stage competes with ongoing work. The UI shows a stage with multiple attempts, FetchFailedException in the logs, and a wall clock unrelated to the data volume. Keeping shuffle files alive past executor death — an external shuffle service, or shuffle tracking under dynamic allocation — is what bounds the damage.

spark.conf.get("spark.dynamicAllocation.shuffleTracking.enabled")
# logs: FetchFailedException, then the parent stage resubmitted as attempt 2

Senior depth: When the failure is deterministic. A single fat partition will kill every executor it lands on, so each retry costs a full stage recompute and gets you no closer. Recognise it from the same task index failing across attempts, stop the retry loop, and fix the partition — salt, skew split, or a smaller partition size — before adding any memory. Raising spark.task.maxFailures in that situation just buys more identical failures.

Common mistake: Expecting the cost of a lost executor to be proportional to its share of the cluster.

Follow-up they will ask: When does retrying make things strictly worse?

Lesson · Simulation

architect

Three of your sixty jobs OOM each month and each one gets a memory bump. What do you change structurally?

Executor OOM · tap to open the answer

Say this first: Require evidence before a memory change, and attack the two real causes — partition sizing and skew — as platform defaults.

Remember these

  • No memory change without the task summary attached
  • Max vs median separates skew from partition sizing from GC
  • Fix defaults: AQE skew join, cores per executor, overhead, storage level
  • Track memory per job as a trend, not per incident

Why: Make a memory increase require the stage's task summary: max versus median Shuffle Read, Peak Execution Memory, and Spill (Disk). If max is far above median it is skew and memory is the wrong lever; if every task is large it is partition sizing; if GC dominates with no spill it is cache or user memory. Then fix the defaults rather than the individual jobs: AQE on with skew join and a sensible advisory partition size, four to five cores per executor with memoryOverhead budgeted for PySpark, MEMORY_AND_DISK as the house storage level, and a target partition size everybody can quote. Track memory per job over time as a metric, because a slow upward creep is the signal that one key or one table is outgrowing the design. The structural failure is not that jobs OOM — it is that the cheapest remediation is also the one that hides the cause.

Senior depth: Per job and per stage: max-to-median task duration ratio, total Spill (Disk), GC time as a share of task time, and configured memory over time. Those four separate the failure classes without opening a single log, and the memory trend line is what catches the slow creep. Alert on the ratio crossing a threshold rather than on the OOM, so you get the ticket before the page.

Common mistake: Letting 'bump the memory' be the one-click remediation, so no job is ever actually diagnosed.

Follow-up they will ask: What would you put on a dashboard for this?

Lesson · Simulation

senior

AQE switched your SMJ to broadcast mid-job. Is that good?

Broadcast joins · tap to open the answer

Say this first: Yes if the build side was small after filters. Bad if stats were wrong and it now OOMs.

Why: AQE join conversion uses runtime size. Watch SQL adaptive UI. If it OOMs, disable conversion for that query or fix the filter so the build side is truly small.

Senior depth: Don't hint a slowly growing dim forever. Partition/filter it, or SMJ when it crosses a documented threshold.

Common mistake: Disabling AQE globally because one query failed.

Follow-up they will ask: How do you cap broadcast size in a lakehouse with growing dims?

Lesson · Simulation

advanced

The threshold is 100 MB, the table is 40 MB of Parquet, and Spark still will not broadcast. Why?

Broadcast joins · tap to open the answer

Say this first: The planner compares against its estimate of the decoded size, and Parquet on disk is compressed and encoded.

Remember these

  • Parquet on disk is compressed; the decision is about decoded size
  • explain("cost") shows the sizeInBytes the planner is using
  • AQE compares against measured shuffle output, not an estimate

Why: Parquet stores columns with dictionary encoding, run-length encoding, and block compression, so 40 MB on disk can be several hundred megabytes once decoded into rows. Spark's sizeInBytes for a file-based relation starts from file size, and plan-level estimates propagate through filters and projections with rules that are often pessimistic — a filter the optimizer cannot reason about leaves the estimate untouched. Read explain("cost") to see the sizeInBytes and rowCount the planner is actually using on that side; if it is wildly wrong, ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS gives it real numbers. The honest alternative is to let AQE decide, because spark.sql.adaptive.autoBroadcastJoinThreshold is evaluated against the measured size of a completed shuffle rather than an estimate. Measure before hinting: the whole failure mode of a hint is asserting a size instead of checking one.

df.explain("cost")  # read sizeInBytes and rowCount on the build side

Senior depth: Run the query and read the BroadcastExchange node in the SQL / DataFrame tab — it reports the data size of the relation it built and the time spent building it. That is the number to put in a design doc. Pair it with the dimension's growth rate so the decision carries an expiry date, rather than a hint that quietly becomes a broadcast of eight gigabytes.

Common mistake: Comparing a table's on-disk footprint directly against autoBroadcastJoinThreshold.

Follow-up they will ask: How do you measure the real broadcast size after the fact?

Lesson · Simulation

senior

A query fails with 'Could not execute broadcast in 300 secs'. Walk the mechanism, then the fix.

Broadcast joins · tap to open the answer

Say this first: The build side's own computation did not finish inside spark.sql.broadcastTimeout — the timeout is the symptom, not the cause.

Remember these

  • BroadcastExchange is its own job; the join blocks on its result
  • The timeout usually reflects slow build-side computation, not network
  • The driver assembles the relation before executors receive it

Why: A BroadcastExchange runs as its own job: the build side is computed, collected to the driver, assembled into a hash relation, and shipped to executors, while the join waits on that future. spark.sql.broadcastTimeout bounds that wait, at 300 seconds by default. Hitting it almost never means the network was slow — it means producing the build side was slow, usually because the 'small' side is a heavy subquery, a wide scan with no pruning, or a JDBC read, or because the driver is struggling to assemble a relation much larger than expected. Open the broadcast job in the SQL tab: its duration and reported data size tell you which. Fixing the build side's plan is the real work, because raising the timeout only lets the driver try longer to collect something it should not be collecting.

spark.conf.set("spark.sql.broadcastTimeout", "600")  # buys time, fixes nothing

Senior depth: When the measured relation size is a meaningful fraction of driver heap or of executor storage memory, or when the build side needs its own expensive plan to produce. At that point a sort-merge join on a well-partitioned key is cheaper and far more stable than a broadcast that is one growth step from an outage. Write the threshold down as a policy with a number, not as a per-query judgement call.

Common mistake: Raising broadcastTimeout as the fix and then being surprised by a driver OOM instead.

Follow-up they will ask: At what point do you stop broadcasting altogether?

Lesson · Simulation

architect

Write the broadcast policy for a lakehouse where dimensions keep growing. What goes in it?

Broadcast joins · tap to open the answer

Say this first: A measured size budget per join, a named owner for every hint, and an alert when a dimension approaches the threshold.

Remember these

  • The budget comes from driver heap and per-executor copies, not from taste
  • Every hint needs a measured size, a growth rate, and a review date
  • Alert on dimension size approaching the threshold, before jobs fail
  • Design dimensions to stay small: current-row views, narrow projections

Why: Start from a number: the maximum broadcast relation size you accept, derived from driver heap and from executor storage memory, since every executor holds a copy. Require that any explicit broadcast hint cites the measured size from the BroadcastExchange node plus the table's growth rate, and carries a review date — a hint is an assertion with an expiry. Keep automatic conversion on and let AQE make the runtime call where it can, because it compares against measured shuffle size rather than a stale estimate. Then monitor: track each dimension's size and alert before it crosses the budget, so the failure arrives as a ticket rather than a nightly outage. Finally make dimensions broadcastable by design — current-row views instead of full SCD history, narrow projections, a sane layout — rather than raising the threshold as data grows.

Senior depth: AQE decides per run from the measured size, so the plan can flip between runs and the cost profile becomes non-deterministic — and a dimension creeping just under the threshold gets broadcast to every executor on a large cluster. Cap spark.sql.adaptive.autoBroadcastJoinThreshold explicitly rather than inheriting it, and treat plan flips as an observable event you record per run.

Common mistake: Codifying a higher autoBroadcastJoinThreshold instead of codifying a size budget and an owner.

Follow-up they will ask: What is the failure mode of letting AQE decide everything?

Lesson · Simulation

senior

AQE coalesced 2000 shuffle partitions down to 12, and now one task is huge. What happened?

Adaptive Query Execution · tap to open the answer

Say this first: Coalesce packed leftover data into too few partitions, or skew was hidden.

Why: Coalesce is for empty/small partitions. If data is uneven, you can get fat tasks. Inspect post-AQE partition sizes. Disable coalesce for that query or salt.

Senior depth: spark.sql.adaptive.enabled and compare SQL UI plans — same query, different later stages.

Common mistake: Blaming AQE without looking at the adaptive plan.

Follow-up they will ask: How do you show AQE on/off in an interview demo?

Lesson · Simulation

advanced

How does AQE decide the final number of shuffle partitions, and which configs bound it?

Adaptive Query Execution · tap to open the answer

Say this first: It packs adjacent shuffle partitions together until each merged one is near the advisory size, within the min and initial partition bounds.

Remember these

  • Coalescing groups contiguous partitions up to the advisory size
  • initialPartitionNum sets the granularity AQE merges from
  • minPartitionNum stops a small query collapsing to one task
  • AQEShuffleRead in the final plan reports the coalescing applied

Why: After the map stage reports per-partition sizes, the coalescing rule walks the partitions in order and groups contiguous ones into a single reducer task whose total is close to spark.sql.adaptive.advisoryPartitionSizeInBytes. spark.sql.adaptive.coalescePartitions.minPartitionNum floors the result so a small query does not collapse to one task, and spark.sql.adaptive.coalescePartitions.initialPartitionNum sets the pre-coalesce count independently of spark.sql.shuffle.partitions — set the initial number high so there is granularity to merge from, and let AQE pick the final number. Because the grouping is contiguous, an uneven size distribution can still produce one fat merged partition, which is the usual explanation for 'AQE coalesced to twelve and now one task is huge'. Verify by reading the post-shuffle partition sizes and the AQEShuffleRead node in the final plan, which reports the coalescing it applied.

spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "128m")
spark.conf.set("spark.sql.adaptive.coalescePartitions.initialPartitionNum", "2000")

Senior depth: The same node reports a skew split instead, showing that a partition was broken into several reader slices fed to separate tasks. Reading that node is how you prove a skew split happened rather than hoping it did: no AQEShuffleRead annotation on the skewed side means the thresholds were never met and the partition went to one task exactly as before.

Common mistake: Tuning spark.sql.shuffle.partitions downward while AQE is on, instead of raising the initial count and setting an advisory size.

Follow-up they will ask: What does AQEShuffleRead look like for a skew split rather than a coalesce?

Lesson · Simulation

advanced

AQE converted a sort-merge join to a broadcast, but the Exchange is still in the final plan. Explain.

Adaptive Query Execution · tap to open the answer

Say this first: The shuffle had already executed — that is how AQE measured the side it decided to broadcast — so the plan reads the existing shuffle output locally.

Remember these

  • AQE measures after the map stage, so the shuffle write already happened
  • The local shuffle reader avoids the network fetch, not the write
  • A runtime conversion saves less than a correctly planned broadcast

Why: AQE's decisions happen after a shuffle map stage completes, so by the time it knows the build side is small those shuffle files already exist. It cannot un-write them, so the converted plan keeps the Exchange and reads its output with a local shuffle reader: each executor reads the blocks it wrote itself, avoiding a network fetch, and the broadcast hash join runs on top. You see this as an Exchange still present with an AQEShuffleRead marked local, controlled by spark.sql.adaptive.localShuffleReader.enabled. The saving is real — the expensive part of a sort-merge join is the sort and the all-to-all fetch, not the map-side write — but smaller than a broadcast planned from the start. That is also why fixing statistics still beats relying on AQE when the size is knowable up front.

spark.conf.get("spark.sql.adaptive.localShuffleReader.enabled")
# final plan: Exchange + AQEShuffleRead local + BroadcastHashJoin

Senior depth: Whenever the size is knowable before runtime and the query runs often. A planned broadcast skips the map-side shuffle write entirely, which on a large fact table is the bulk of the cost. AQE is the safety net for cases estimates cannot reach — UDF results, JDBC sources, heavily filtered intermediates — not a reason to leave ANALYZE TABLE undone on your core tables.

Common mistake: Reading the leftover Exchange as proof the conversion never happened.

Follow-up they will ask: So when is fixing statistics still worth the effort?

Lesson · Simulation

senior

You want reproducible plans for a benchmark. Is disabling AQE the right call?

Adaptive Query Execution · tap to open the answer

Say this first: For one isolated experiment, defensible; as a standing configuration, no — you would be benchmarking a plan production never runs.

Remember these

  • AQE makes plans data-dependent by design
  • Benchmarking without it measures a plan production never runs
  • Capture the final plan per run and treat plan flips as a result

Why: AQE makes plans data-dependent, so two runs over slightly different data can produce different joins and partition counts, which is genuinely awkward for a controlled comparison. For a single experiment, pinning spark.sql.adaptive.enabled to false on both arms is fine as long as you report it. But production runs with AQE on, so a benchmark without it measures a plan that will not be used, and the differences are not small — coalescing changes task counts, conversion changes join strategy, skew splitting changes stage time. The better approach is to keep AQE on and capture the final plan per run from the SQL tab or the event log, then compare like with like and treat a plan change as a result rather than as noise. Disabling AQE to make a dashboard look stable is how teams end up tuning a configuration that does nothing in production.

spark.conf.set("spark.sql.adaptive.enabled", "false")  # experiment only, and report it

Senior depth: Scope it to the feature and the query. If a runtime broadcast conversion caused an executor OOM, cap spark.sql.adaptive.autoBroadcastJoinThreshold rather than killing adaptive execution; if coalescing produced a fat task, set an advisory partition size or disable spark.sql.adaptive.coalescePartitions.enabled for that query only. Global off is a blunt instrument that also surrenders the skew handling you probably depend on.

Common mistake: Turning AQE off permanently after one query regressed, instead of capping the specific feature involved.

Follow-up they will ask: Which single AQE feature would you disable instead of the whole thing?

Lesson · Simulation

Practice by topic