Spark beginner interview questions
Partitions, actions, lazy evaluation, and what the driver actually does.
Question 1 of 123
Walk a Spark action from code to rows without skipping a layer.
Answer it out loud, then reveal. Play steps through like the simulators.
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 123 of 123 questions.
beginnerWalk a Spark action from code to rows without skipping a layer.
What happens when Spark executes a job? · tap to open the answer
Say this first: Code → logical plan → job → stages → tasks → (optional shuffle) → result.
Why: Transformations are lazy. The action builds a plan, splits it at shuffles into stages, runs one task per partition, then returns a small result or writes files.
Common mistake: Saying 'Spark runs the DataFrame line by line'.
Follow-up they will ask: Where does a shuffle sit in that chain?
intermediateWhy does one action create multiple stages?
What happens when Spark executes a job? · tap to open the answer
Say this first: Each wide transformation (shuffle) is a stage boundary.
Why: Narrow steps (filter, map, broadcast join) stay in one stage. groupBy, join-without-broadcast, repartition cut a new stage because data must move.
Common mistake: Counting jobs instead of stages when debugging runtime.
Follow-up they will ask: How do you see stage boundaries in the physical plan?
beginnerAt which point in the chain does Spark decide how many tasks a stage gets?
What happens when Spark executes a job? · tap to open the answer
Say this first: At the leaf for scan stages — from file splits and spark.sql.files.maxPartitionBytes — and at the Exchange for every stage after a shuffle.
Remember these
- Scan stage task count comes from file splits and maxPartitionBytes
- openCostInBytes is why tiny files inflate task counts
- Post-shuffle stages take their count from the Exchange partitioning
- AQE coalescing can reduce the post-shuffle count at runtime
Why: For the first stage, the number of tasks equals the number of input partitions the file source produced. Spark packs file splits into partitions up to spark.sql.files.maxPartitionBytes and charges each file an opening cost of spark.sql.files.openCostInBytes, which is what makes a thousand tiny files produce a thousand mostly-idle tasks instead of one sensible one. For stages after a shuffle, the count comes from the Exchange's partitioning, which defaults to spark.sql.shuffle.partitions. With AQE enabled, that post-shuffle count is then revised at runtime: small adjacent partitions are coalesced toward spark.sql.adaptive.advisoryPartitionSizeInBytes, so the Stages page can show fewer tasks than the configured number. You can read all of this off the UI without guessing — each stage's task count is on the Stages page, and the Exchange nodes are in the SQL / DataFrame tab's plan.
df.rdd.getNumPartitions() # scan-side partitions
spark.conf.get('spark.sql.shuffle.partitions') # post-shuffle defaultCommon mistake: Believing spark.sql.shuffle.partitions controls how many tasks read the source files.
Follow-up they will ask: Why can the Stages page show 48 tasks when shuffle partitions is set to 200?
intermediateThe SQL / DataFrame tab shows one query, but the Jobs tab lists three jobs for it. Where did the extra jobs come from?
What happens when Spark executes a job? · tap to open the answer
Say this first: A single query can submit several jobs: broadcast exchanges, AQE shuffle materialisation, and limit scale-up each run as their own job.
Remember these
- One SQL query ID can own several job IDs
- BroadcastExchange collects the small side as its own job
- AQE re-plans after materialising each shuffle, adding jobs
- The SQL tab lists the job IDs belonging to a query
Why: A query ID in the SQL tab is one logical execution, but the scheduler submits jobs whenever it needs a result before continuing to plan or execute. A BroadcastExchange has to collect the small side and build a hash relation before the join can run, and that collection is its own job. With spark.sql.adaptive.enabled, Spark executes shuffle map stages, reads their real output statistics, and only then finalises the downstream plan — so each materialisation boundary appears as a separate job under the same query, which is why the plan root reads AdaptiveSparkPlan with isFinalPlan=true only at the end. A limit can add more jobs through scale-up retries when the first few partitions do not yield enough rows. Click the query in the SQL tab and it lists the associated job IDs, which is the fastest way to attribute them rather than guessing from the Jobs list.
Common mistake: Counting jobs to count user actions, then concluding the notebook ran a cell twice.
Follow-up they will ask: How would you tell an AQE-added job from a second user action?
beginnerWhen does Spark actually read the table?
What happens when Spark executes a job? · tap to open the answer
Say this first: On an action — not when you write filter or groupBy.
Why: The lazy plan is a recipe. show(), count(), write(), collect() cook it. Until then Spark has not scanned storage.
Common mistake: Wrapping every line in count() 'to be safe'.
Follow-up they will ask: What does an unresolved logical plan still not know?
intermediateWhy can Catalyst rearrange your filters after you wrote them in a certain order?
What happens when Spark executes a job? · tap to open the answer
Say this first: The optimizer is allowed to push predicates and prune partitions.
Why: Your code order is not the physical order. Predicate pushdown and partition pruning happen in analysis/optimization. That is why a late filter can still skip files.
Common mistake: Assuming Python line order is the execution order.
Follow-up they will ask: How do you prove a filter was pushed into the scan?
beginnerWhat is a Spark job, exactly?
What happens when Spark executes a job? · tap to open the answer
Say this first: The work triggered by one action.
Why: Each show/count/write/collect is a job (sometimes more with AQE). A job is not a Databricks Workflow. A job contains stages.
Common mistake: Calling the whole notebook 'one Spark job'.
Follow-up they will ask: How many jobs does df.cache(); df.count(); df.count() typically create?
intermediateWhy can AQE turn one job into extra stages at runtime?
What happens when Spark executes a job? · tap to open the answer
Say this first: AQE can coalesce, switch join strategy, and split skew after it sees sizes.
Why: The first stages run, statistics update, later stages change. Spark UI shows SQL / AQE details. That is expected, not a second user action.
Common mistake: Thinking extra stages mean the user clicked twice.
Follow-up they will ask: Which AQE feature changes a shuffle join into a broadcast?
beginnerWhat splits a job into stages?
What happens when Spark executes a job? · tap to open the answer
Say this first: A shuffle — a wide dependency.
Why: Narrow transformations pipeline inside a stage. When keys must co-locate (groupBy, sort merge join, repartition), Spark inserts an Exchange and a new stage.
Common mistake: Saying each transformation is its own stage.
Follow-up they will ask: Is a filter a new stage?
intermediateStage time is 20 minutes but most tasks finished in 30 seconds. What is the stage time?
What happens when Spark executes a job? · tap to open the answer
Say this first: The slowest task — the straggler.
Why: Stages wait for all tasks. Median is a vanity metric. Read max task duration, spill, and shuffle read on that task.
Common mistake: Optimizing average task time while one key holds the stage.
Follow-up they will ask: Which UI chart shows this immediately?
beginnerWhat is the difference between a partition and a task?
What happens when Spark executes a job? · tap to open the answer
Say this first: A partition is a slice of data. A task is the unit of work that reads that slice in a stage.
Why: One partition → one task in that stage. Executors run many tasks over time. Do not call an executor a task.
Common mistake: Using partition, task, and executor as synonyms.
Follow-up they will ask: If you have 200 partitions and 10 executors, how many tasks in a narrow stage?
intermediateWhy can 10 000 partitions make a small table slower?
What happens when Spark executes a job? · tap to open the answer
Say this first: Task scheduling overhead dwarfs the work. Each task has launch cost.
Why: Tiny files / too many partitions → thousands of 20 ms tasks. Spark UI shows high scheduler delay. Coalesce or compact files.
Common mistake: repartition(10000) 'for more parallelism' on a 200 MB table.
Follow-up they will ask: How do you pick a partition count for a 200 GB shuffle?
beginnerWhat comes back to the driver after a shuffle?
What happens when Spark executes a job? · tap to open the answer
Say this first: Only what the action asked for — a few show() rows, a count, or nothing for write.
Why: Shuffle moves data between executors. It does not dump the table on the driver. write() commits files; the driver stays small. collect() is the action that pulls.
Common mistake: Believing every shuffle returns the full dataset to the notebook.
Follow-up they will ask: Does saveAsTable send rows through the driver?
intermediateWhy is shuffle both a network problem and a disk problem?
What happens when Spark executes a job? · tap to open the answer
Say this first: Map tasks write shuffle files locally; reducers fetch over the network; spill hits disk again if a reducer partition is fat.
Why: Spark UI: shuffle write, shuffle read, fetch wait, spill memory/disk. A healthy shuffle is balanced. A sick one has one reducer reading 10× bytes.
Common mistake: Only looking at CPU when a stage is an Exchange.
Follow-up they will ask: What is shuffle partition size aiming at?
beginnerWhat is Apache Spark, in one sentence a hiring manager wants?
What is Apache Spark? · tap to open the answer
Say this first: A distributed engine that processes large datasets in parallel across a cluster.
Why: Spark splits data into partitions, runs a task per partition on executors, and only ships results when an action runs. It is not a database and it does not store your lake.
Common mistake: Calling Spark a 'database' or saying it always holds data in memory.
Follow-up they will ask: What is lazy about a DataFrame transformation?
intermediateWhy can a 20-line notebook do nothing for minutes, then explode when you call show()?
What is Apache Spark? · tap to open the answer
Say this first: Transformations build a plan. Actions execute it.
Why: groupBy / filter / join are lazy. Spark waits for show, count, write, collect. That is when jobs, stages, and tasks appear in Spark UI.
Common mistake: Thinking Spark is slow at parse time because the notebook looks busy.
Follow-up they will ask: Name three actions that trigger a job.
beginnerA teammate says 'Spark is faster than MapReduce because it keeps everything in memory.' What is wrong with that sentence?
What is Apache Spark? · tap to open the answer
Say this first: Spark avoids writing intermediate results to the cluster filesystem between steps — it does not hold your dataset in RAM.
Remember these
- MapReduce wrote to HDFS between steps; Spark pipelines inside a stage
- Shuffle map output always lands on executor local disk
- Working memory is spark.memory.fraction; overflow spills to disk
- Caching is opt-in with persist(), not the default behaviour
Why: MapReduce materialised the output of every map and every reduce to HDFS, so a five-step job paid five durable round trips. Spark pipelines narrow operators inside a single stage and fuses them into one generated Java method, so rows pass through filters and projections without a write in between. Memory still matters, but only as working space governed by spark.memory.fraction: when a sort or aggregation does not fit, the task spills sorted runs to the executor's local disk and you see it in the Spill (Memory) and Spill (Disk) columns on the Stages page. Shuffle output is always written to local disk by the map tasks, in every Spark version — there is no in-memory shuffle. Caching a dataset in RAM is opt-in via persist(), and it is a separate decision from how the engine executes. So the honest sentence is 'fewer durable writes and operator fusion', not 'in memory'.
from pyspark import StorageLevel
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count() # materialises the cache; check the Storage tabCommon mistake: Repeating the marketing line '100x faster, in-memory' and then being unable to explain the Spill (Disk) column.
Follow-up they will ask: If memory is not the whole story, what does whole-stage codegen actually fuse?
intermediateYour manager asks why a 40 GB daily aggregation needs Spark instead of pandas on one big VM. Argue both sides.
What is Apache Spark? · tap to open the answer
Say this first: Spark wins on spill, parallel scan, and recovery; a single large VM often wins on latency and cost at 40 GB.
Remember these
- Parquet expands 3-5x in memory, so one-process sizing targets the worst day
- Spark gives partitioned scan, disk spill, and per-task retry
- Spark charges startup, a driver JVM, and a shuffle per groupBy
- Default shuffle partition count makes small jobs scheduler-bound
Why: pandas loads the whole frame into one process, and the in-memory footprint of a 40 GB Parquet file is routinely three to five times the file size once strings and indexes are expanded, so the VM has to be sized for the worst day, not the average one. Spark reads the same table as many partitions sized by spark.sql.files.maxPartitionBytes, runs one task per partition, and spills to local disk instead of dying when a partition does not fit. It also retries a failed task up to spark.task.maxFailures instead of losing the whole run. Against that, Spark charges you cluster startup, a driver JVM, and a shuffle for every groupBy — and the default spark.sql.shuffle.partitions will cut a small aggregation into tiny tasks whose scheduling delay dominates their work, visible as a Duration histogram piled near zero on the Stages page. At 40 GB the deciding factors are usually growth rate and whether the output has to feed a lake table that other jobs read. Pick Spark when the data or the fan-out is growing; pick the VM when the job is stable and single-reader.
Common mistake: Answering 'Spark scales' with no number, and never mentioning that 200 shuffle partitions on 40 GB is mostly overhead.
Follow-up they will ask: What would you set spark.sql.shuffle.partitions to for this job, and how would you arrive at the number?
beginnerName the three Spark cluster roles and what each one must not do.
Spark cluster architecture · tap to open the answer
Say this first: Driver coordinates. Executors compute. Cluster manager places the JVMs.
Why: The driver builds the DAG and tracks tasks. Executors run tasks and hold cache. The cluster manager (YARN, K8s, Databricks) allocates machines. Big data stays on executors and storage — not in the driver heap.
Common mistake: Saying the driver 'processes the data'.
Follow-up they will ask: Where does collect() put the result?
intermediateIf the driver dies, why does the whole job die even if executors are healthy?
Spark cluster architecture · tap to open the answer
Say this first: The driver owns the SparkSession, DAG, and task scheduler. Executors are workers, not the brain.
Why: Lose the driver JVM and there is no one to retry tasks or return the action result. Databricks job clusters fail the run; notebooks disconnect.
Common mistake: Restarting one executor to 'fix a driver OOM'.
Follow-up they will ask: What Spark UI page is served from the driver?
beginnerOn Databricks you never run spark-submit. So who starts the driver, and which machine is it?
Spark cluster architecture · tap to open the answer
Say this first: The control plane asks the cloud for VMs; the driver node is one of them, and your notebook attaches to a SparkSession already running there.
Remember these
- Control plane requests VMs; driver and executors run in your data plane
- Attaching a notebook joins an existing driver, it does not start one
- Each notebook gets its own SparkSession on the shared driver
- The Spark UI is served by the driver process
Why: When you create a cluster, the Databricks control plane issues the instance requests into your account's data plane, then starts a driver JVM on the driver node and executor JVMs on the worker nodes. Attaching a notebook does not create a Spark application — it opens a REPL against the driver that is already live, which is why a second notebook on the same cluster shares executors but gets its own SparkSession. The driver node also hosts the Python process behind your PySpark code, so notebook variables and anything you collect live on that machine, next to but outside the JVM heap. A single-node cluster simply runs executors in the driver's process, which is fine for teaching and wrong for anything with real volume. The Spark UI you open is served by the driver, so losing the driver loses the UI too.
Common mistake: Describing a notebook attach as 'submitting a Spark application' and then being unable to say why two notebooks see each other's executors.
Follow-up they will ask: Two notebooks on one cluster both run a wide job. What do they contend for?
intermediateYour spark-submit works from the edge node but the identical scheduled run fails. What does deploy mode have to do with it?
Spark cluster architecture · tap to open the answer
Say this first: In client mode the driver runs on the edge node; in cluster mode it runs on a worker, so anything local to the edge node disappears.
Remember these
- Client mode driver = your submitting host; cluster mode driver = a worker container
- Local paths and localhost services vanish in cluster mode
- Ship files with --files and reference them by bare filename
- Client mode ties application lifetime to the submitting shell
Why: With --deploy-mode client the driver JVM is your edge-node process, so it can read local config files, resolve hostnames on that box, and reach a service on localhost. With --deploy-mode cluster the cluster manager launches the driver inside a container on some worker, and a path like /etc/app/creds.json or a jdbc:...@localhost URL is now a FileNotFoundException or a connection refused. The fix is to ship dependencies explicitly with --files and --jars so they land in the container working directory, and to reference them by filename rather than absolute path. Client mode also keeps the driver tied to your shell: closing the terminal kills the application, and all result traffic crosses the edge node's network link. Cluster mode is the right default for scheduled work precisely because the driver's lifetime and locality stop depending on a human session.
spark-submit --deploy-mode cluster \
--files /etc/app/creds.json \
job.py
# inside the driver: open('creds.json')Common mistake: Assuming a job is environment-independent because it ran interactively on the gateway host.
Follow-up they will ask: Where do you read the driver logs in each mode?
beginnerWhat is an RDD, and why do we still mention it?
RDDs: lineage and partitions · tap to open the answer
Say this first: The original distributed collection. DataFrames compile down toward RDD/Tungsten execution.
Why: You rarely write RDD code now. Interviewers want: partitions, lineage, and that DataFrames are the API you should use.
Common mistake: Starting a new ETL in RDD map/reduce.
Follow-up they will ask: What is lineage on an RDD?
intermediateWhy is Dataset/DataFrame usually faster than hand-rolled RDDs?
RDDs: lineage and partitions · tap to open the answer
Say this first: Catalyst optimizes the plan; Tungsten uses off-heap/binary rows. RDDs are Java objects.
Why: RDD map on Row objects boxes everything. DataFrame expressions can whole-stage codegen. You also get predicate pushdown on files.
Common mistake: rdd.filter because 'it's more control'.
Follow-up they will ask: When is an RDD the right tool?
beginnergroupByKey or reduceByKey for a per-key sum on an RDD, and why?
RDDs: lineage and partitions · tap to open the answer
Say this first: reduceByKey — it combines values on the map side, so far fewer bytes cross the shuffle.
Remember these
- reduceByKey combines on the map side before the shuffle
- groupByKey ships every raw value and can overwhelm one reducer
- Compare Shuffle Write bytes to see the difference
- DataFrame groupBy inserts partial aggregation automatically
Why: groupByKey ships every value for a key to the reducer that owns it and hands you an iterable, so the shuffle carries the raw dataset and one hot key means one reducer holding all of its values in memory. reduceByKey applies your associative function locally within each map partition first, so a partition with ten thousand rows for a key sends one partial sum instead of ten thousand values. Shuffle Write bytes in the UI show the difference directly, usually by an order of magnitude on skewed data. The combine step is also what keeps a single large key from blowing up a reducer, because the per-key payload is already reduced before it travels. aggregateByKey and combineByKey are the general forms for when the accumulator type differs from the value type. The DataFrame equivalent is a groupBy with an aggregate, where Catalyst inserts the partial aggregation for you — one more reason not to hand-roll this in RDDs.
rdd.reduceByKey(lambda a, b: a + b) # partial sums shuffle
# rdd.groupByKey().mapValues(sum) # whole dataset shufflesCommon mistake: Using groupByKey because the code reads more naturally, then blaming skew for the reducer OOM.
Follow-up they will ask: When is groupByKey genuinely the right choice?
intermediateWhat does a single df.rdd.map(...) cost you in the middle of a DataFrame pipeline?
RDDs: lineage and partitions · tap to open the answer
Say this first: It converts optimised binary rows into JVM or Python objects, ends codegen, and leaves everything downstream outside Catalyst's reach.
Remember these
- DeserializeToObject converts binary rows into real objects
- Codegen fusion ends at the RDD boundary
- Pushdown and pruning cannot cross back over the conversion
- PySpark adds pickling and a python worker hop per row
Why: Inside the DataFrame API, rows are Tungsten InternalRow in off-heap-friendly binary form and operators are fused by whole-stage codegen. Calling .rdd forces a DeserializeToObject step that materialises each row as a real object, which in PySpark additionally pickles it and sends it to a python worker, and the plan shows the codegen island ending right there. Everything after the conversion is opaque to the optimizer: no predicate pushdown into the scan, no column pruning based on later usage, and no Photon or vectorised execution on that branch. Converting back with toDF or createDataFrame restarts the optimizer from a plan whose input is an RDD, so pushdown cannot reach the original files at all. The practical consequence is that one convenience line can turn a pruned, fused scan into a full read plus per-row serialisation in both directions. If you need row-level logic, express it as column functions, or as a pandas_udf when it genuinely must be Python.
df.rdd.map(lambda r: (r.id, r.amount * 2)).toDF(['id', 'amt']).explain()
# DeserializeToObject ... MapPartitions ... no PushedFilters on the sourceCommon mistake: Dropping to .rdd for one transformation and expecting the rest of the plan to optimise as before.
Follow-up they will ask: How would you express that same map as a column expression?
beginnerWhat is a DataFrame in Spark?
DataFrames and schemas · tap to open the answer
Say this first: A distributed table with a schema and a lazy query plan.
Why: Rows are partitioned across executors. Operations return a new plan, not a local pandas object. spark.createDataFrame on a huge Python list still starts on the driver.
Common mistake: Treating it like pandas (df[i] loops, collect to 'feel the data').
Follow-up they will ask: How do you print the schema without executing a job?
intermediateWhen should you drop to RDDs from a DataFrame?
DataFrames and schemas · tap to open the answer
Say this first: Almost never for ETL. DataFrames get Catalyst and Tungsten. RDDs skip that.
Why: Use RDDs only for custom partitioning or APIs that still need them. groupBy/join/filter belong on DataFrames/Datasets.
Common mistake: rdd.map for a column expression Catalyst could optimize.
Follow-up they will ask: What do you lose when you call rdd.map on a DataFrame?
beginnerdf2 = df.withColumn('x', lit(1)) gives you a new DataFrame. Did Spark copy the data?
DataFrames and schemas · tap to open the answer
Say this first: No — it appended a node to a query plan. No rows exist yet for either variable.
Remember these
- A DataFrame is a handle to an immutable plan, not a data container
- Transformations append plan nodes; no rows are materialised
- df2's plan contains df's plan, so both actions re-run the shared part
- Immutability is what makes Catalyst rewrites safe
Why: A DataFrame is a handle to an immutable logical plan, so every transformation returns a new handle wrapping the old plan plus one operator. Nothing is allocated per row, which is why chaining fifty transformations on a petabyte-scale table costs microseconds. The two variables are not two datasets: df2's plan contains df's plan as a subtree, and if you run an action on each one, Spark executes the shared part twice unless you cache it. Immutability is what lets Catalyst rewrite freely — it can reorder a filter below a projection because no one holds a mutable reference to an intermediate state. Call df2.explain() and you see a Project node over whatever df was; call df2.count() and only then does a job appear.
df2 = df.withColumn('x', lit(1))
df2.explain() # Project over the original plan, zero jobsCommon mistake: Reasoning about DataFrames like pandas objects and worrying about the memory cost of 'another copy'.
Follow-up they will ask: If both df and df2 are used in actions, what have you paid twice?
intermediateTwo teams union their DataFrames and the output has amounts in the region column. Both schemas 'match'. What happened?
DataFrames and schemas · tap to open the answer
Say this first: union() matches columns by position, not by name, so a reordered upstream schema silently swaps values.
Remember these
- union() aligns by position; only count and type compatibility are checked
- unionByName() aligns by name and fails loudly on mismatch
- allowMissingColumns=True fills nulls — useful, and easy to misuse
- The Union plan node looks identical under both semantics
Why: DataFrame.union is the SQL UNION ALL contract: it lines up the Nth column of the left side with the Nth column of the right side and only checks that the count matches and the types are compatible. If both sides happen to have a string and a decimal in different orders, or if an upstream table added a column in the middle, the plan is still valid and no error is raised. Use unionByName when the inputs are produced independently; it resolves by column name and fails loudly on a mismatch. With allowMissingColumns=True it fills absent columns with null instead of failing, which is the right tool for schema evolution but hides genuinely missing data if you reach for it reflexively. The plan node is Union either way, so you cannot tell from the physical plan which semantics you got — only the API call tells you. Compare df.columns on both sides in a test and the bug never ships.
a.unionByName(b) # strict, name-aligned
a.unionByName(b, allowMissingColumns=True) # evolution-tolerantCommon mistake: Assuming union validates names because the first version of both tables happened to be in the same order.
Follow-up they will ask: How would a unit test catch this before production?
intermediateYou filter a Parquet table on amount > 100. How do you prove the filter reached the scan instead of running above it?
DataFrames and schemas · tap to open the answer
Say this first: Read explain('formatted') and check PushedFilters on the FileScan node, then confirm Input Size / Records dropped in the UI.
Remember these
- FileScan prints PartitionFilters, PushedFilters, and DataFilters separately
- Pushdown skips row groups via Parquet min/max statistics
- Casts, functions, and UDFs on the column block pushdown
- The observable proof is a drop in Input Size / Records
Why: The formatted plan prints three separate lists on a file scan: PartitionFilters for predicates that eliminate whole directories, PushedFilters for predicates handed to the Parquet reader, and DataFilters for predicates Spark still evaluates itself. A predicate in PushedFilters lets the reader skip row groups whose min/max statistics cannot match, so the win shows up as reduced Input Size / Records on the Stages page, not as a changed plan shape. If the predicate appears only in a separate Filter node above the scan, every row was read and discarded. Common reasons pushdown fails: the predicate wraps the column in a function or a cast, it compares against a non-deterministic expression, it is an OR spanning pushable and non-pushable terms, or the whole thing is hidden inside a UDF that the optimizer cannot see into. Verify spark.sql.parquet.filterPushdown is still on in the Environment tab before blaming the query. The two-step habit — plan first, then Input Size — is what turns a guess into evidence.
df.filter('amount > 100').explain('formatted')
# FileScan parquet ... PushedFilters: [IsNotNull(amount), GreaterThan(amount,100)]Common mistake: Declaring pushdown works because the query got faster, without checking Input Size / Records.
Follow-up they will ask: Why does a filter on cast(id as string) usually not push down?
beginnerTwo teammates write the same aggregation — one in spark.sql(), one in the DataFrame API. Which one runs faster?
Spark SQL: the same engine, a different keyboard · tap to open the answer
Say this first: Neither. Both go through Catalyst and end up as the same physical plan.
Why: A SQL string and a chain of DataFrame calls both become an unresolved logical plan, then get analyzed, optimized, and planned by the same code path. Run explain('formatted') on both and you see the same FileScan, Exchange, and HashAggregate nodes.
Common mistake: Claiming the DataFrame API is 'closer to the engine' so it skips a layer that costs runtime.
Follow-up they will ask: So what would actually make the two versions differ?
intermediateYour notebook created a temp view; the next notebook on the same cluster can't see it. Explain.
Spark SQL: the same engine, a different keyboard · tap to open the answer
Say this first: createOrReplaceTempView is session-scoped, and the other notebook has its own SparkSession.
Why: Temp views live in the session catalog and disappear with the session. createGlobalTempView registers into the global_temp database, shared across sessions of the same application, and must be referenced as global_temp.name. Neither is in the metastore — only saveAsTable creates something another job can find tomorrow.
Common mistake: Assuming a temp view is a table because SELECT * worked in the same notebook.
Follow-up they will ask: When would you reach for a global temp view instead of just writing a table?
beginnerWhat does CACHE TABLE do that CREATE OR REPLACE TEMP VIEW does not?
Spark SQL: the same engine, a different keyboard · tap to open the answer
Say this first: A temp view registers a name over a plan; CACHE TABLE materialises the data into executor storage and runs immediately.
Remember these
- A temp view is a name over a plan; referencing it re-executes it
- CACHE TABLE materialises an InMemoryRelation and is eager by default
- CACHE LAZY TABLE defers until the first read
- A cache hit shows as InMemoryTableScan in the physical plan
Why: A temp view is a catalog entry pointing at a logical plan, so referencing it three times means executing it three times. CACHE TABLE takes the same plan, executes it, and stores the result as an InMemoryRelation held by the executors' block managers, which then appears on the Storage tab with a Fraction Cached figure. It is eager by default — the statement itself triggers the job — whereas CACHE LAZY TABLE defers until the first query touches it. After caching, the physical plan of a query over that name shows InMemoryTableScan instead of a FileScan, which is the one-glance confirmation that the cache is being used. UNCACHE TABLE releases it; so does losing the session, because both views and caches are session-scoped and are not metastore objects.
CREATE OR REPLACE TEMP VIEW busy AS SELECT * FROM orders WHERE region = 'US';
CACHE TABLE busy;
SELECT count(*) FROM busy; -- InMemoryTableScanCommon mistake: Calling a temp view a cache and being puzzled that three queries over it cause three scans.
Follow-up they will ask: When is caching that view the wrong move?
intermediateAfter a runtime upgrade your SQL pipeline starts throwing arithmetic overflow and cast errors on data it accepted for years. What changed?
Spark SQL: the same engine, a different keyboard · tap to open the answer
Say this first: ANSI SQL mode turns silently-null results into errors for invalid casts, overflow, and division by zero.
Remember these
- ANSI mode turns invalid casts, overflow, and divide-by-zero into errors
- Error classes are explicit: CAST_INVALID_INPUT, ARITHMETIC_OVERFLOW
- try_cast, try_divide, try_sum restore lenient behaviour per expression
- Verify the effective setting in the Environment tab, not from memory
Why: With spark.sql.ansi.enabled off, Spark's historical behaviour is permissive: cast('abc' as int) yields null, an integer overflow wraps or nulls, and x / 0 returns null. With ANSI mode on, each of those raises instead — CAST_INVALID_INPUT, ARITHMETIC_OVERFLOW, DIVIDE_BY_ZERO — so the first run after the flag flips surfaces every latent data quality problem at once. This is a correctness improvement and an operational shock, which is why it should be rolled out deliberately rather than inherited from a runtime default. The migration path is per-expression, not global: try_cast, try_divide, try_add, and try_sum give you the lenient result explicitly, so the code states which nulls are intentional. Check the Environment tab to confirm the effective value before arguing about behaviour, since cluster defaults and session settings both contribute. Keeping ANSI mode on with explicit try_* calls is the version that still catches real bugs.
SELECT try_cast(raw_qty AS INT) AS qty,
try_divide(revenue, units) AS unit_price
FROM bronze.sales;Common mistake: Turning ANSI mode back off globally instead of marking the handful of expressions that are meant to be lenient.
Follow-up they will ask: How would you find every expression that depends on lenient casting before flipping the flag?
beginnerWhat is a transformation versus an action?
Narrow vs wide transformations · tap to open the answer
Say this first: Transformation: new DataFrame, lazy. Action: job, side effect or result.
Why: filter, select, groupBy, join are transformations. show, count, write, collect, take are actions. groupBy alone does not shuffle until an action.
Common mistake: Saying groupBy 'runs the shuffle immediately'.
Follow-up they will ask: Is cache a transformation or an action?
intermediateNarrow vs wide transformation — give one example of each and the cost.
Narrow vs wide transformations · tap to open the answer
Say this first: Narrow: filter/map — no shuffle. Wide: groupBy/join — shuffle and a new stage.
Why: Narrow tasks read only their partition. Wide tasks wait on Exchange. Broadcast join is wide in API but not a shuffle of the fact table.
Common mistake: Calling join always a shuffle join.
Follow-up they will ask: Why can a filter after a groupBy not reduce shuffle bytes of that groupBy?
beginnerdf.drop('notes') versus df.select('id', 'amount') — which one changes what gets read from storage?
Narrow vs wide transformations · tap to open the answer
Say this first: Neither, directly. The optimizer derives the scan's required columns from the whole plan, so both end up pruning the same way.
Remember these
- Column pruning is a plan-level rule, not a property of drop or select
- The FileScan output schema in explain() shows what is actually read
- Columnar formats skip unreferenced columns on disk
- select is an allow-list; drop is a deny-list that passes new columns through
Why: Column pruning is a Catalyst rule, not an API behaviour: it walks the plan, collects the columns any operator actually needs, and pushes that set down into the file scan. Because Parquet and Delta are columnar, the reader then skips the unreferenced columns on disk entirely. So whether you expressed the intent as a drop or as a select, the FileScan's output schema is the same and you can read it right off explain('formatted'). What does defeat pruning is referencing the column after all — a UDF that takes the whole row, a select('*') late in the chain, or a write that happens to carry everything. The difference between drop and select is maintenance, not performance: select is an allow-list that fails when a required column disappears, drop is a deny-list that silently carries new upstream columns through.
df.drop('notes').explain('formatted')
# FileScan parquet [id#1,amount#2] -- notes never readCommon mistake: Claiming drop is wasteful because 'Spark reads the column first and then throws it away'.
Follow-up they will ask: Which one would you prefer in a pipeline whose source schema evolves?
intermediateIs df.orderBy('amount') narrow or wide, and why does it sometimes create an extra job?
Narrow vs wide transformations · tap to open the answer
Say this first: It is wide — a global sort range-partitions the data, and building the range boundaries requires sampling the input first.
Remember these
- Global sort uses Exchange rangepartitioning plus a local Sort
- Boundary selection needs a sampling job before the shuffle
- A dominant value makes one range band an enormous task
- orderBy().limit(n) becomes TakeOrderedAndProject, not a full sort
Why: A global ordering cannot be produced by sorting each partition independently, so Spark range-partitions: partition 0 gets the lowest values, partition 1 the next band, and so on, after which a local sort per partition yields a globally ordered result. To choose those band boundaries it has to know the value distribution, so it samples the input in a separate job before the shuffle — that is the short job you see appear ahead of the real work. The physical plan shows Exchange rangepartitioning followed by Sort, which is structurally different from the hashpartitioning Exchange a groupBy produces. Skew hurts a range partitioner directly: if one value dominates, its band becomes one enormous task, and the sort spills. When you only need ordering inside groups, sortWithinPartitions avoids the global shuffle entirely, and when you only need the top rows, orderBy().limit(n) is planned as TakeOrderedAndProject, which keeps a bounded heap per partition instead of sorting everything.
df.orderBy('amount').limit(10).explain() # TakeOrderedAndProject
df.sortWithinPartitions('amount') # no global shuffleCommon mistake: Assuming sort is narrow because each partition 'just sorts itself'.
Follow-up they will ask: Why is sortWithinPartitions enough before a window function in some cases?
intermediatedistinct(), dropDuplicates(['id']), and groupBy('id').agg(max('ts')) all deduplicate. Pick one and defend it.
Narrow vs wide transformations · tap to open the answer
Say this first: distinct() dedupes on every column, dropDuplicates on a subset keeps an arbitrary row, and only an explicit aggregate or window makes the surviving row deterministic.
Remember these
- distinct() groups by all columns, including ones you did not mean
- dropDuplicates on a subset keeps an arbitrary, run-dependent row
- An explicit aggregate or window makes the survivor deterministic
- Window dedupe costs an Exchange plus a Sort but keeps whole rows
Why: distinct() is shorthand for grouping by all columns, so it shuffles on the full row and plans as a HashAggregate with every column as a key — expensive and wrong if one column is a load timestamp that differs between duplicates. dropDuplicates(['id']) shuffles on id only and keeps whichever row arrives first within each partition, which means the winner can change between runs and after a retry. If which duplicate survives matters — and for late-arriving updates it always does — state the rule explicitly: groupBy the key and aggregate the fields, or use row_number() over a window partitioned by the key and ordered by the version column, then filter to rank one. The window form costs an Exchange plus a Sort and is the one to reach for when you need whole rows rather than a few aggregated fields. Whatever you choose, check the HashAggregate keys in the plan match the key you meant.
w = Window.partitionBy('id').orderBy(col('updated_at').desc())
dedup = df.withColumn('rn', row_number().over(w)).filter('rn = 1').drop('rn')Common mistake: Using dropDuplicates on a key and then being surprised that a rerun picks a different version of the row.
Follow-up they will ask: How would you dedupe deterministically without a window function?
beginnerName four Spark actions and what each returns to the driver.
Actions that trigger jobs · tap to open the answer
Say this first: count → a number. show → a few printed rows. collect → all rows. write → nothing (files on storage).
Why: take/limit also pull a small result. foreach runs on executors. The dangerous one is collect on a large frame.
Common mistake: Using collect() as the default way to 'see data'.
Follow-up they will ask: Which action is safest to check a pipeline ran?
intermediateWhy can count() be expensive even though it returns one integer?
Actions that trigger jobs · tap to open the answer
Say this first: It still executes the full plan, including shuffles.
Why: count after a wide transformation pays the shuffle. It is a cheap result, not a cheap job. Sometimes Spark can optimize count on a scan; not after a join.
Common mistake: Sprinkling count() after every transform in production.
Follow-up they will ask: When is approx_count_distinct the better interview answer?
beginnerdf.first() returns instantly on a 2 TB table. Why is that not evidence that the plan is fast?
Actions that trigger jobs · tap to open the answer
Say this first: A limit pushes into the plan, so Spark reads only enough partitions to produce one row and stops.
Remember these
- first() is take(1), planned as CollectLimit over a few partitions
- Input Size / Records shows how little was actually read
- A limit above an aggregate still pays for the aggregate
- Under a selective filter, the scan expands partition-set by partition-set
Why: first() is take(1), and take is planned as a CollectLimit: Spark runs the job against a small number of partitions, checks whether it has enough rows, and only scans more if it does not. On a wide, unfiltered table the very first partition supplies the row, so the action touches a fraction of a percent of the data and Input Size / Records in the UI confirms it. That tells you nothing about a full scan, and it tells you nothing about the shuffle a groupBy in the same plan would cost, because a limit above an aggregate still has to finish the aggregate. The behaviour also inverts under a selective filter: if almost no partition contains a matching row, Spark keeps expanding how many partitions it scans — governed by spark.sql.limit.scaleUpFactor — and the cheap-looking action becomes a near-full scan across several jobs. Use count() or an explicit aggregate when you want to know what the plan actually costs.
df.first() # cheap: one partition is usually enough
df.filter('rare_flag').first() # may expand to most of the tableCommon mistake: Using a fast first() or show() as a smoke test for whether the pipeline's real scan will meet its SLA.
Follow-up they will ask: Why can the same first() be slow on a heavily filtered table?
intermediateYou need to know how many rows a job wrote. Why not just call count() on the DataFrame before writing?
Actions that trigger jobs · tap to open the answer
Say this first: count() re-executes the whole plan, and the number it returns is not necessarily what the write produced; read the write's own metrics instead.
Remember these
- count() before write() executes the plan a second time in full
- The two executions can disagree if the source or plan is non-deterministic
- Delta DESCRIBE HISTORY exposes numOutputRows for the committed version
- The write stage's Output Size / Records is the UI equivalent
Why: write() is an action, so a count() before it is a second, independent execution of the same plan — you pay every scan and shuffle twice. Worse, the two executions can disagree whenever the source is still changing or the plan contains anything non-deterministic, so the count becomes a number you log rather than a number you can trust. The writer already records what it did: on Delta, DESCRIBE HISTORY exposes operationMetrics with numOutputRows and numFiles for the version your job committed, and the Spark UI reports Output Size / Records for the write stage. For a programmatic hook that works across formats, register a QueryExecutionListener and read the executed plan's output-rows metric after each query. If you need a validation count for a quality gate, run it against the written table afterwards — then you are measuring the artefact rather than a second guess at it.
df.write.mode('append').saveAsTable('silver.orders')
spark.sql('DESCRIBE HISTORY silver.orders').select('version', 'operationMetrics').show(1, False)Common mistake: Logging a pre-write count as the row count of the write, and doubling the pipeline's cost to get it.
Follow-up they will ask: How would you make that count a hard quality gate without a third execution?
beginnerWhat does lazy evaluation mean for a DataFrame?
Why Spark is lazy · tap to open the answer
Say this first: Spark records the recipe and waits for an action.
Why: That lets Catalyst optimize the whole chain (predicate pushdown, join reorder) instead of running each line. It also means errors can appear late.
Common mistake: Evaluating each transformation when the line runs, like pandas.
Follow-up they will ask: How do you force execution without collect()?
intermediateYou persist a DataFrame, then add a filter, then count. Why is cache unused?
Why Spark is lazy · tap to open the answer
Say this first: The cached plan is the unfiltered one; the new plan does not match, or you never materialized the cache.
Why: cache() is lazy. Need an action to populate Storage. A new filter is a different lineage unless you cache after the filter.
Common mistake: Assuming persist survives arbitrary extra operators.
Follow-up they will ask: How do you confirm a scan hit cache in Spark UI?
beginnerWhich of these does work right now: df.printSchema(), df.explain(), df.limit(5), df.count()?
Why Spark is lazy · tap to open the answer
Say this first: printSchema and explain are driver-only metadata operations, limit is another lazy transformation, and only count submits a job.
Remember these
- printSchema reads a resolved plan; no stage is submitted
- explain does driver-side planning work but submits no job
- limit() is a transformation producing LocalLimit and GlobalLimit
- count() is the only action here and runs the whole plan
Why: printSchema reads the already-resolved schema off the logical plan, so it costs nothing beyond what resolution already did at read time. explain runs Catalyst through analysis and planning and prints the result, which is real driver work but submits no stage — it is the tool to reach for when you want to inspect a plan without paying for it. limit(5) is a transformation: it appends a LocalLimit and GlobalLimit pair and returns a new DataFrame, which is why df.limit(5) on its own prints nothing and shows nothing in the Jobs tab. count() is the only action in the list, and it executes the entire plan including any shuffle, then returns a single number. Knowing which of the four is free is what lets you debug a plan on a production-sized table without launching anything.
df.printSchema() # free
df.explain('formatted') # driver only
df.limit(5) # still lazy
df.count() # Job submittedCommon mistake: Thinking limit(5) is an action because show(5) looks similar and does run.
Follow-up they will ask: Which one would you use to check a filter pushed down, on a 10 TB table?
intermediateYou called cache() then count(). The Storage tab says the DataFrame is 55% cached. What happens on the next query?
Why Spark is lazy · tap to open the answer
Say this first: Cached blocks are served from memory or local disk; the uncached partitions are recomputed from the source every time you read.
Remember these
- DataFrame cache() is MEMORY_AND_DISK, so overflow spills rather than drops
- Memory-only levels recompute uncached partitions from lineage
- Storage tab Fraction Cached and Size on Disk distinguish the two
- Cache competes with execution memory and can cause shuffle spill
Why: cache() on a DataFrame uses the MEMORY_AND_DISK storage level, so when the storage pool fills, blocks are written to the executor's local disk rather than dropped — a partially cached relation usually means memory pressure pushed part of it to disk, and reads of those blocks pay disk I/O instead of a recompute. With an explicitly memory-only level the behaviour is harsher: blocks that do not fit are simply not cached, and every read recomputes them from lineage, which on an expensive upstream plan can make the cached version slower than no cache at all. Either way the Storage tab's Fraction Cached and Size on Disk columns tell you which situation you are in, and the plan shows InMemoryTableScan with the InMemoryRelation underneath. Caching also competes for the same unified pool as execution memory, bounded below by spark.memory.storageFraction, so a large cache can push your shuffles into spilling. If the relation does not fit, the right move is usually to cache a narrower projection or a filtered subset, or to write an intermediate table instead.
df.select('id', 'amount').filter('amount > 0').cache().count()
# cache a narrow, filtered projection rather than the whole tableCommon mistake: Caching a wide table and counting the cache as free, while the shuffle in the same job starts spilling.
Follow-up they will ask: How would you decide between caching this and writing an intermediate table?
beginnerWhat is a Spark partition?
Partitions: the unit of parallelism · tap to open the answer
Say this first: A chunk of a DataFrame that one task reads in a stage.
Why: Input partitions often follow files. After a shuffle, spark.sql.shuffle.partitions (or AQE) decides how many. One partition = one task, not one executor.
Common mistake: One partition per executor as a rule.
Follow-up they will ask: Who decides input partitions for a Parquet scan?
intermediaterepartition vs coalesce — when do you use each?
Partitions: the unit of parallelism · tap to open the answer
Say this first: repartition shuffles to a new count. coalesce reduces without a full shuffle (can stay unbalanced).
Why: repartition(n) for even parallelism or a join key. coalesce(n) after a filter that dropped most rows, to avoid a shuffle. Don't coalesce to 1 on a huge frame except for a tiny output.
Common mistake: coalesce(1) to 'make a single CSV' on a 400 GB table.
Follow-up they will ask: What does partitionBy on write control versus RDD partitions?
beginnerYou point Spark at a folder of 500 small Parquet files and getNumPartitions() returns 68, not 500. Where did 68 come from?
Partitions: the unit of parallelism · tap to open the answer
Say this first: The file scan packs small files together into splits until a split reaches spark.sql.files.maxPartitionBytes.
Remember these
- Input partitions come from byte-packed splits, not from file count
- spark.sql.files.maxPartitionBytes caps the bytes in one input split
- openCostInBytes charges each tiny file so they pack instead of fragment
- One large splittable file can become many partitions
Why: Spark does not map one file to one partition. The scan planner walks the file list and bin-packs files into splits, charging each file an opening cost of spark.sql.files.openCostInBytes on top of its real size, and closes a split once it reaches spark.sql.files.maxPartitionBytes. So 500 files of a few MB each collapse into far fewer, fatter input partitions. The opposite also happens: one 3 GB splittable Parquet file becomes many partitions because a single file can be cut at row-group boundaries. If you want the real number, read it off the DataFrame with getNumPartitions or look at the task count on the FileScan stage in the Spark UI.
print(spark.conf.get("spark.sql.files.maxPartitionBytes"))
df = spark.read.parquet("/mnt/raw/events")
print(df.rdd.getNumPartitions())Common mistake: Saying 'one file equals one partition' and then being unable to explain the 68.
Follow-up they will ask: Which of those two settings would you change to get more parallelism on the scan?
intermediateYour scan stage runs 40 tasks on a cluster with 160 cores. A teammate raises spark.sql.shuffle.partitions to 800 and nothing changes. Why not?
Partitions: the unit of parallelism · tap to open the answer
Say this first: spark.sql.shuffle.partitions only sizes post-shuffle Exchanges; the scan's partition count is decided by file splitting.
Remember these
- Input sizing and shuffle sizing are different knobs on different stages
- Parallelism in a stage is capped by that stage's partition count
- Lower spark.sql.files.maxPartitionBytes to split a scan finer
- repartition after the scan works but costs a full shuffle
Why: There are two completely separate sizing paths. Input partitions are produced by the FileScan from spark.sql.files.maxPartitionBytes and the file layout, and no shuffle config touches them. Shuffle partitions are the reducer count that an Exchange targets, so spark.sql.shuffle.partitions only takes effect downstream of a groupBy, join, or repartition. With 40 input partitions and 160 cores, 120 cores sit idle for the whole scan stage no matter what the shuffle setting says. The fixes are to lower maxPartitionBytes so the files split finer, or to insert an explicit repartition right after the scan and accept the shuffle that buys you parallelism.
spark.conf.set("spark.sql.files.maxPartitionBytes", 32 * 1024 * 1024)
df = spark.read.parquet("/mnt/raw/events")
print(df.rdd.getNumPartitions())Common mistake: Treating spark.sql.shuffle.partitions as a global parallelism dial for the whole job.
Follow-up they will ask: If the files are gzipped CSV instead of Parquet, does lowering maxPartitionBytes still help?
beginnerrepartition(200) and coalesce(200) both leave me with 200 partitions. What is actually different?
Repartition vs coalesce vs partitionBy · tap to open the answer
Say this first: repartition does a full shuffle and evens out the data; coalesce merges existing partitions with no shuffle and keeps them uneven.
Why: repartition inserts an Exchange — round-robin, or hash if you pass columns — so partitions come out roughly equal in size. coalesce is a narrow operation that glues partitions together on the same executor, which is cheap but inherits whatever imbalance the input had.
Common mistake: Trying to use coalesce to increase the partition count — it can only reduce.
Follow-up they will ask: When is the uneven result of coalesce perfectly fine?
intermediateSomeone put coalesce(1) before the write to get a single output file, and now the entire pipeline runs single-threaded. Explain the mechanism.
Repartition vs coalesce vs partitionBy · tap to open the answer
Say this first: coalesce has no shuffle boundary, so the reduced partition count pushes back and the upstream transformations run with one task too.
Why: Because coalesce does not create an Exchange, the join or aggregation feeding the write ends up inside the same stage at one partition — one task on one executor doing everything. repartition(1) adds an Exchange instead, so upstream work keeps its parallelism and only the final write is serial.
Common mistake: Believing coalesce only affects the write because that is where the call sits in the code.
Follow-up they will ask: So how do you actually produce one output file without wrecking the job?
beginnerA DataFrame has 200 partitions and you call coalesce(400). What do you end up with?
Repartition vs coalesce vs partitionBy · tap to open the answer
Say this first: Still 200 — coalesce can only reduce, and asking for more is silently ignored.
Remember these
- coalesce only reduces partition count; increasing it is a silent no-op
- No Exchange appears in the plan, so there is no visual clue
- repartition(n) is the only way to raise the count
- Confirm with getNumPartitions rather than trusting the call
Why: coalesce merges existing partitions into fewer, larger ones inside the same stage; it has no mechanism to split a partition because splitting requires redistributing rows, and redistributing rows is a shuffle. So coalesce(400) on a 200-partition frame returns a frame with 200 partitions and raises no error, which is exactly why the bug survives code review. The plan shows no Exchange at all, so there is nothing in explain output to catch your eye either. If you genuinely want 400 partitions you must call repartition(400), which inserts Exchange RoundRobinPartitioning(400) and evens the sizes out.
print(df.rdd.getNumPartitions()) # 200
print(df.coalesce(400).rdd.getNumPartitions()) # 200
print(df.repartition(400).rdd.getNumPartitions()) # 400Common mistake: Believing coalesce failed loudly somewhere, when it quietly returned the frame unchanged.
Follow-up they will ask: Why can repartition increase the count when coalesce cannot?
intermediateYou added repartition(200, 'customer_id') to fix a slow stage and one task is still carrying everything. Why did repartition not spread the data?
Repartition vs coalesce vs partitionBy · tap to open the answer
Say this first: Hash partitioning on a key sends every row with that key to one partition, so a single hot key stays one fat task.
Remember these
- repartition(n) is round-robin and even; repartition(n, col) is hash and co-located
- One hot key always lands in exactly one hash partition
- Raising n does not split a single key across partitions
- Pass columns only when a downstream operator needs that key co-located
Why: repartition(n) with no columns is Exchange RoundRobinPartitioning, which distributes rows blindly and therefore evenly. repartition(n, col) is Exchange hashpartitioning(col, n), which guarantees co-location of equal keys, and co-location is the opposite of balance when one key dominates. If customer_id 0 or a sentinel like 'UNKNOWN' holds 40% of the rows, that 40% lands in exactly one of the 200 partitions no matter how large n gets. The Stage page shows it immediately: one task with enormous Shuffle Read and a large Spill (Disk) while the other 199 finish in seconds. The fix is salting the hot key, letting AQE skew handling split it, or dropping the key argument if you only wanted balance and never needed co-location.
df.repartition(200) # even, no co-location
df.repartition(200, "customer_id") # co-located, skew preserved
df.groupBy("customer_id").count().orderBy(F.desc("count")).show(5)Common mistake: Passing the join key to repartition when the real goal was balanced task sizes.
Follow-up they will ask: How would you confirm which key is hot without a full scan of the data?
beginnerWhat does a shuffle do?
What a shuffle actually does · tap to open the answer
Say this first: Moves records so equal keys land on the same reducer. Disk + network + a stage boundary.
Why: Map tasks write shuffle files. Reducers fetch their slice. groupBy, distinct, join (non-broadcast), and repartition all shuffle.
Common mistake: Thinking shuffle is 'Spark being slow' rather than a specific Exchange.
Follow-up they will ask: Which Spark UI numbers prove a shuffle?
intermediateWhy can a shuffle take longer than the compute?
What a shuffle actually does · tap to open the answer
Say this first: Network fetch, disk spill, and waiting on the slowest reducer.
Why: Shuffle write/read bytes, fetch wait, spill. A skewed key makes one reducer read most of the data. Compression and AQE help only if the plan is sane.
Common mistake: Adding CPU cores when fetch wait is the bottleneck.
Follow-up they will ask: What is a shuffle block?
intermediateA stage reads a 40 GB table and the Spark UI reports 300 GB of Shuffle Write. How is the shuffle bigger than the input?
What a shuffle actually does · tap to open the answer
Say this first: Shuffle Write measures the serialized rows leaving the map side after your transformations, not the bytes you read.
Remember these
- Shuffle Write is serialized post-map output, not input bytes
- Compare Shuffle Write Records to Input Records to separate width from multiplication
- collect_list and exact distinct skip map-side partial aggregation
- Projecting away unused columns before the Exchange shrinks the shuffle directly
Why: Three things commonly inflate it. First, Parquet input is compressed and column-pruned on disk, while shuffle rows are full materialized records, so decompressed-and-widened data is simply bigger. Second, anything that multiplies rows before the Exchange — an explode, a many-to-many join, a cross-product of flags — ships the multiplied result. Third, aggregations that cannot do a map-side partial merge, like collect_list or an exact count distinct over a high-cardinality key, send every raw row to the reducer instead of a partial aggregate. Compare the stage's Input Size / Records with its Shuffle Write Records: if the record count grew, you have a row-multiplication problem; if only the bytes grew, you have a width and serialization problem and the fix is to select fewer columns before the Exchange.
(orders.select("customer_id", "amount") # drop 30 unused columns first
.groupBy("customer_id")
.sum("amount"))Common mistake: Comparing Shuffle Write to the compressed table size on storage and concluding Spark is broken.
Follow-up they will ask: Which aggregate functions can do a map-side partial merge and which cannot?
beginnerBroadcast join vs shuffle join — when do you pick each?
Join strategies in Spark · tap to open the answer
Say this first: Broadcast if one side fits in memory on every executor. Shuffle (sort-merge) if both sides are large.
Why: Broadcast replicates the small table. SMJ partitions both by key and sorts. A wrong broadcast of a 'small' 10 GB table OOMs executors.
Common mistake: Broadcasting the fact table because 'joins should be broadcast'.
Follow-up they will ask: How do you force a broadcast in Spark SQL?
intermediateThe plan says SortMergeJoin but you expected broadcast. Why?
Join strategies in Spark · tap to open the answer
Say this first: Statistics said both sides were large, or AQE/broadcast threshold was below the build side.
Why: Check table stats, spark.sql.autoBroadcastJoinThreshold, and filters that Catalyst didn't push (so size is wrong). AQE can switch later if enabled.
Common mistake: Hints without checking stats.
Follow-up they will ask: What happens if stats are stale after a 10× load?
beginnerList the join strategies Spark can pick and say what rules each one out.
Join strategies in Spark · tap to open the answer
Say this first: Broadcast hash, shuffle hash, sort-merge, and the nested-loop fallbacks — size, hints, and whether there is an equi-key decide which survives.
Remember these
- Broadcast wins when a side fits under autoBroadcastJoinThreshold or is hinted
- SortMergeJoin is the general fallback: shuffle both sides, sort, merge
- No equi-condition means nested loop or Cartesian, which is usually a bug
- AQE can change the strategy after the shuffle reports real sizes
Why: For an equi-join, the planner prefers BroadcastHashJoin when one side is hinted or its estimated size is under spark.sql.autoBroadcastJoinThreshold, because replicating the small side removes the shuffle on the big one. If neither side is broadcastable, it considers a shuffle hash join when one side is substantially smaller and spark.sql.join.preferSortMergeJoin allows it, then falls back to SortMergeJoin, which shuffles and sorts both sides and is the safe general case. Without an equality condition there is nothing to hash or sort on, so you get BroadcastNestedLoopJoin or a Cartesian product, and seeing either of those in a plan for a join you believed was keyed means your condition is not an equi-condition. AQE can revise the choice at runtime once real shuffle sizes are known, which is why the plan you read before execution may not be the plan that ran.
Common mistake: Naming only broadcast and sort-merge, then being stuck when the plan shows BroadcastNestedLoopJoin.
Follow-up they will ask: What in your SQL would produce a BroadcastNestedLoopJoin by accident?
intermediateYou put a broadcast hint on the left side of a LEFT OUTER JOIN and the plan ignores it. Why?
Join strategies in Spark · tap to open the answer
Say this first: For a left outer join the left side must stream, so only the right side is a legal build side.
Remember these
- Hash join build side determines which outer semantics are possible
- LEFT OUTER can broadcast only the right side; RIGHT OUTER only the left
- FULL OUTER cannot use a broadcast hash join at all
- An illegal hint is dropped silently, not reported as an error
Why: A hash join streams one side and probes a hash table built from the other. Outer semantics constrain which is which: a LEFT OUTER JOIN must emit every left row exactly once, including unmatched ones, and that is only possible if the left side is the streaming side and the right side is the build side. So Spark can broadcast the right side of a left outer join and the left side of a right outer join, and for a FULL OUTER JOIN neither side qualifies for a broadcast hash join at all. A hint that violates this is not an error — the planner simply cannot honour it and falls back to SortMergeJoin, which is why your hint vanished silently. If the small table really is on the left, flip the join to a RIGHT OUTER JOIN with the sides swapped, or restructure to an inner join plus a separate anti-join if the semantics allow.
big.join(F.broadcast(small), "customer_id", "left") # legal: small is the right side
small.join(F.broadcast(big), "customer_id", "left") # hint ignored, falls back to SMJCommon mistake: Concluding the hint syntax is wrong when the join type is what made it impossible.
Follow-up they will ask: Your plan shows BroadcastHashJoin but the build side is the table you thought was huge. What should you check?
beginnerWhat does cache() actually do?
Cache and persist · tap to open the answer
Say this first: Marks the DataFrame to be stored after the next action. It is lazy.
Why: First action computes and stores partitions (MEMORY_AND_DISK by default for DataFrames). Second action can skip recomputation if the plan matches and data still fits.
Common mistake: cache() as an action that runs immediately.
Follow-up they will ask: How do you uncache?
intermediateWhen is cache the wrong answer?
Cache and persist · tap to open the answer
Say this first: When you read the data once, or when the cached set is bigger than memory and thrashes disk.
Why: Cache for reuse in the same session (ML iterative, branching QA). For a single write, cache adds memory pressure. Checkpoint if lineage is huge.
Common mistake: Caching every intermediate table in a 30-step ETL.
Follow-up they will ask: MEMORY_ONLY vs MEMORY_AND_DISK — which fails harder?
beginnerYou call df.cache() and the next cell is just as slow as before. What was missed?
Cache and persist · tap to open the answer
Say this first: cache() only marks the plan; nothing is stored until an action materializes the partitions.
Remember these
- cache() is lazy; the first action pays and populates
- count() materializes every partition, show() may populate only some
- The Storage tab is empty until materialization happens
- persist with no subsequent action does nothing at all
Why: cache() and persist() register the DataFrame's analyzed plan as an InMemoryRelation and return immediately — no data moves. The first action after that computes the plan and fills the cache as a side effect, so that action pays full price and only later ones can benefit. Which action you use matters too: count() touches every partition and therefore populates all of them, while show() or limit(10) may only compute enough partitions to satisfy the limit, leaving the cache partially filled. The Storage tab is the proof — a cached entry appears only once something materialized it, and Fraction Cached tells you how much of it landed. A persist with no following action is simply a no-op that cost you a line of code.
gold.cache()
gold.count() # this action fills the cache
gold.groupBy("region").count().show() # this one reads itCommon mistake: Describing cache() as an action that moves data at the moment it is called.
Follow-up they will ask: Does unpersist() also wait for an action?
intermediateThe Storage tab shows 'Fraction Cached 63%' for your DataFrame. What does the next action actually do?
Cache and persist · tap to open the answer
Say this first: It reads the 63% from the block manager and recomputes the other 37% from lineage.
Remember these
- Caching is per-partition; missing partitions recompute from lineage
- MEMORY_ONLY drops partitions that do not fit; MEMORY_AND_DISK spills them
- Eviction is LRU across everything cached in the session
- Select and filter before caching to shrink what you are asking to store
Why: Caching is per-partition, not all-or-nothing. A partition is stored only if it fits where the storage level says it should go, so with MEMORY_ONLY any partition that did not fit is simply dropped, and with MEMORY_AND_DISK it spills to the executor's local disk instead of being lost. Either way the next action reads the cached partitions from the block manager and re-executes the full lineage for the missing ones, which is why a partly cached frame can be barely faster than an uncached one. A fraction below 100% usually means the cached set is bigger than the storage memory available, or that other cached frames in the same session evicted yours under LRU. The remedies are to cache less — project columns away and filter before caching — to switch to MEMORY_AND_DISK, or to unpersist frames you no longer need.
from pyspark import StorageLevel
narrow = gold.select("customer_id", "region", "amount")
narrow.persist(StorageLevel.MEMORY_AND_DISK)
narrow.count()Common mistake: Reading a partial fraction as 'cache is working' and never checking why 37% is missing.
Follow-up they will ask: Two notebooks on the same cluster both cache large frames. What happens?
beginnerSame rows, same cluster: why does the Parquet version of a table read so much faster than the CSV version?
File formats: why Parquet reads less · tap to open the answer
Say this first: Parquet is columnar, typed, and carries statistics — a CSV scan has to read and parse every byte of every line.
Why: A Parquet scan reads only the columns in your select and can skip entire row groups using min/max statistics. CSV has no schema, no column boundaries, and no stats, so Spark parses the whole file. The plan shows FileScan parquet with a ReadSchema listing just the columns you asked for.
Common mistake: Saying Parquet is faster 'because it's compressed' and stopping there.
Follow-up they will ask: What is still a fair use case for CSV or JSON?
intermediateYou read a CSV with inferSchema=true and a job appears in Spark UI before your action ever runs. What is it?
File formats: why Parquet reads less · tap to open the answer
Say this first: Schema inference is an eager extra pass over the data to guess column types.
Why: inferSchema reads the file (all of it for CSV by default) to decide types, which submits its own job ahead of your transformation. Passing an explicit StructType removes that pass; Parquet and ORC never need it because the schema is in the file footer.
Common mistake: Blaming that first slow cell on cluster warm-up instead of the inference job.
Follow-up they will ask: What else can inferSchema get wrong besides being slow?
beginnerDescribe the physical layout of a Parquet file from the top down.
File formats: why Parquet reads less · tap to open the answer
Say this first: A file is a sequence of row groups; each row group holds one contiguous chunk per column, split into pages, with a footer carrying the schema, offsets, and statistics.
Remember these
- Row group, then column chunk, then page — that is the nesting
- The footer carries schema, chunk offsets, and per-chunk min/max/null stats
- Column pruning is a ranged read of only the chunks you asked for
- Row-group skipping uses footer statistics, not an index
Why: At the top a Parquet file is divided into row groups, each covering a horizontal slice of rows. Inside a row group each column is stored as its own contiguous column chunk, and each chunk is divided into pages that carry the encoded values plus definition and repetition levels for nulls and nested fields. The footer at the end of the file holds the schema and, for every column chunk, its byte offset and statistics such as min, max, and null count. This is why a reader opens the footer first: it learns the schema with no inference pass, issues ranged reads for only the column chunks in ReadSchema, and drops whole row groups whose min/max cannot satisfy the pushed filter. Grouping like values together is also what makes dictionary encoding and run-length encoding effective before the block codec runs.
Common mistake: Describing Parquet as 'columns stored separately' without row groups, which makes splittability impossible to explain.
Follow-up they will ask: Given that layout, why is a Parquet file splittable but a gzipped CSV is not?
intermediateA read with mergeSchema=true sits for eleven minutes before a single row is processed. What is it doing?
File formats: why Parquet reads less · tap to open the answer
Say this first: It is opening the footer of every file in the table to union their schemas before planning can finish.
Remember these
- mergeSchema reads every file footer, not a sample
- Cost scales with file count, which is why small files make it unbearable
- An explicit StructType removes the resolution step entirely
- Delta and Iceberg keep the schema in the log, so it is one metadata read
Why: Without mergeSchema, Spark reads one file's footer (or the summary metadata) and assumes the rest match, so planning is cheap. With mergeSchema=true it must read every footer and reconcile the field sets, which on a table of hundreds of thousands of objects is hundreds of thousands of small remote reads before any job you care about starts. You see it as a long pause in the driver, or as a schema-inference job in the Jobs tab that precedes your real work. The fix is to stop asking the file layer to resolve schema at all: pass an explicit StructType, or move to a table format like Delta or Iceberg where the current schema lives in the log and resolution is a single metadata read. Keep mergeSchema as a one-off tool for investigating a messy landing zone, not as a standing option on a production read.
from pyspark.sql.types import StructType, StructField, StringType, DoubleType
schema = StructType([StructField("id", StringType()), StructField("amount", DoubleType())])
df = spark.read.schema(schema).parquet("/mnt/raw/events")Common mistake: Leaving mergeSchema on permanently 'in case the schema changes' and paying for it on every single run.
Follow-up they will ask: What happens when two files genuinely disagree on a column's type?
beginnerYou need a per-customer running total but the report still needs every transaction row. Window or groupBy?
Window functions: keep the row, add the answer · tap to open the answer
Say this first: Window — it computes over related rows but returns a value per row, so nothing collapses.
Why: groupBy + agg reduces each key to one row; a window function keeps input cardinality and adds a column. In the plan you get a WindowExec with a Sort under it instead of a HashAggregate, and no join-back step to reattach the aggregate.
Common mistake: Doing groupBy then joining the aggregate back to the detail — two shuffles for what a window does in one.
Follow-up they will ask: How would you use a window to grab the latest row per key?
intermediaterowsBetween(-2, 0) and rangeBetween(-2, 0) on the same ordered window return different numbers. Why?
Window functions: keep the row, add the answer · tap to open the answer
Say this first: rowsBetween counts physical rows; rangeBetween compares values of the ordering column.
Why: rowsBetween is positional — the two preceding rows regardless of their values. rangeBetween is a value frame: every row whose ordering value falls inside the offset is included, so gaps shrink the frame and ties pull in all peers. With duplicate ordering values, rangeBetween rows share identical frames.
Common mistake: Using rangeBetween on a timestamp and expecting exactly three rows in the frame.
Follow-up they will ask: Which one do you need for a true 7-day rolling sum over daily data with missing days?
beginnersum('amount').over(Window.partitionBy('region')) gives every row in a region the same number. Is that broken?
Window functions: keep the row, add the answer · tap to open the answer
Say this first: No — with no orderBy the frame is the entire partition, so every row correctly gets the partition total.
Remember these
- No orderBy means the frame is the whole partition
- Adding orderBy switches the default frame to a running total
- sum().over() with no orderBy is legal and silently returns a total
- row_number() does throw without orderBy, which builds a false expectation
Why: A window has three parts and only partitionBy was supplied. Without an orderBy there is no notion of position, so the frame defaults to the whole partition and each row's sum covers every row with the same region — the partition total, broadcast onto every row. Add orderBy and the default frame changes to unbounded preceding through current row, which is what turns the same expression into a running total. This is the most dangerous window mistake precisely because it does not raise an error: the query succeeds and the numbers look plausible. row_number() behaves differently, it refuses to plan without an orderBy, which is why people assume sum() would too.
w_total = Window.partitionBy("region")
w_running = Window.partitionBy("region").orderBy("ts")
df.withColumn("region_total", F.sum("amount").over(w_total)) \
.withColumn("running", F.sum("amount").over(w_running))Common mistake: Treating the repeated value as a bug and adding a distinct or a groupBy to 'fix' it.
Follow-up they will ask: Where is a partition total on every row actually the thing you want?
intermediatelag(amount, 1) returns null for the first row of every partition, and you also need the previous non-null value rather than the previous row. How do you handle both?
Window functions: keep the row, add the answer · tap to open the answer
Say this first: Pass a default to lag for the boundary, and use last(col, ignorenulls=True) over a frame ending before the current row for the previous non-null.
Remember these
- lag never crosses a partition boundary, so a boundary null is correct
- lag's third argument supplies a default value in place of the null
- Previous non-null needs last(..., ignorenulls=True) over a trailing frame
- Multiple functions over the same WindowSpec share one Exchange and Sort
Why: The null at the partition boundary is correct behaviour: there is no previous row inside that partition, and lag never crosses a partitionBy boundary. lag takes a third argument that substitutes a default, so lag('amount', 1, 0) gives zero instead of null where you want a numeric baseline rather than a gap. 'Previous non-null' is a different question — it is not a fixed offset, so lag cannot express it. The idiom is last with ignorenulls over a frame of unboundedPreceding to -1, which scans backwards through the ordered partition and returns the most recent value that was not null. Both operators require an orderBy, and both pay the same Exchange and Sort beneath WindowExec, so computing them together over one WindowSpec costs no more than computing one.
w = Window.partitionBy("device_id").orderBy("ts")
df.withColumn("prev", F.lag("reading", 1, 0.0).over(w)) \
.withColumn("last_known", F.last("reading", ignorenulls=True).over(w.rowsBetween(Window.unboundedPreceding, -1)))Common mistake: Filling boundary nulls with a coalesce to zero when the business meaning is 'no prior reading'.
Follow-up they will ask: What does last(..., ignoreNulls=True) do if the entire preceding frame is null?
beginnerYou wrote a Python UDF to uppercase a column. Why does a reviewer push back?
UDFs: the box Catalyst cannot see into · tap to open the answer
Say this first: Every row leaves the JVM for a Python process and comes back; upper() never leaves the JVM at all.
Why: A Python UDF serializes rows to a Python worker, evaluates them one at a time, and serializes results back — it shows up as BatchEvalPython in the plan. Built-ins like upper, regexp_replace, and to_timestamp compile into whole-stage codegen and operate on Tungsten binary rows directly.
Common mistake: Assuming the UDF is fine 'because it's just string formatting'.
Follow-up they will ask: Name a built-in you would use instead of a UDF you have written before.
intermediateYou moved a filter into a UDF and the scan went from 4 GB to 900 GB. Explain.
UDFs: the box Catalyst cannot see into · tap to open the answer
Say this first: Catalyst cannot see inside a UDF, so the predicate is not pushed into the scan and no partitions get pruned.
Why: A UDF is an opaque expression: it can never become a PushedFilter or a PartitionFilter, so Spark reads every file and filters afterwards above BatchEvalPython. explain('formatted') shows PushedFilters empty and the FileScan's Input Size / Records covering the whole table.
Common mistake: Blaming 'Python is slow' when the real cost is the extra 896 GB of I/O.
Follow-up they will ask: How do you keep the custom logic but restore pruning?
beginnerYou registered your UDF with spark.udf.register and call it from SQL instead of the DataFrame API. Does that change how it runs?
UDFs: the box Catalyst cannot see into · tap to open the answer
Say this first: No — registration only makes it callable by name in SQL; the execution path and the optimizer blindness are identical.
Remember these
- Registration affects name resolution, not the execution path
- The plan still shows BatchEvalPython either way
- Catalyst remains unable to push filters through the function
- SQL registration widens the blast radius by making it easy to reuse
Why: spark.udf.register adds the function to the session's function registry so a SQL string can resolve the name. The resulting expression is the same opaque Python UDF node, so the plan still shows BatchEvalPython, rows are still pickled out to a Python worker and back, and Catalyst still cannot push a predicate through it or fold it into whole-stage codegen. The only real difference is who can call it: analysts writing SQL can now use it, which in practice makes the performance problem spread further rather than shrink. Compare it with a built-in SQL function, which resolves to a Catalyst Expression with a codegen implementation and lives entirely inside the JVM.
spark.udf.register("clean_phone", clean_phone, StringType())
spark.sql("SELECT clean_phone(phone) FROM customers").explain("formatted")Common mistake: Believing a SQL-registered UDF gets optimized like a built-in because it is written in SQL syntax.
Follow-up they will ask: Is a Spark SQL function defined with CREATE FUNCTION in SQL the same thing?
intermediateYour plan shows BatchEvalPython and a colleague's shows ArrowEvalPython. What changed in the code, and what does it change at runtime?
UDFs: the box Catalyst cannot see into · tap to open the answer
Say this first: They moved to a pandas_udf, so rows cross to Python as Arrow columnar batches instead of one pickled row at a time.
Remember these
- BatchEvalPython crosses per row; ArrowEvalPython crosses per batch
- pandas_udf receives and returns a pandas Series, so the body vectorizes
- maxRecordsPerBatch controls how many rows cross at a time
- Arrow makes it faster, not transparent — pushdown is still blocked
Why: BatchEvalPython is the plain Python UDF path: each row is serialized with pickle, written over a socket to a Python worker, evaluated by CPython, and serialized back. ArrowEvalPython is what a pandas_udf plans to: Spark converts a batch of rows into an Arrow record batch, hands the whole batch over once, and your function receives a pandas Series and returns a pandas Series. You pay the boundary crossing per batch instead of per row, and the function body runs vectorized in NumPy or pandas rather than in a Python loop. Getting there means declaring the function with pandas type hints and a pandas_udf decorator; the relevant knobs are spark.sql.execution.arrow.pyspark.enabled and spark.sql.execution.arrow.maxRecordsPerBatch, which sets how many rows go over at once. What does not change is that it is still an opaque expression — nothing pushes through an ArrowEvalPython node either.
import pandas as pd
from pyspark.sql.functions import pandas_udf
@pandas_udf("double")
def to_usd(amount: pd.Series, rate: pd.Series) -> pd.Series:
return amount * rateCommon mistake: Writing a pandas_udf whose body still loops row by row over the Series, which keeps the Python cost and only saves serialization.
Follow-up they will ask: What would make you raise or lower maxRecordsPerBatch?
beginnerWhat runs on the driver versus an executor?
Driver vs executors · tap to open the answer
Say this first: Driver: SparkSession, Catalyst, DAG, task scheduling. Executor: tasks, cache, shuffle files.
Why: Your notebook code until an action is driver-side. After the action, work is split into tasks that executors run on partitions. Results come back only if the action asks (show, collect).
Common mistake: Thinking each executor has its own SparkSession you should create.
Follow-up they will ask: Why is creating a SparkSession inside a foreach a bug?
intermediateWhy does write() not send the table through the driver?
Driver vs executors · tap to open the answer
Say this first: Executors write partitions in parallel to storage. The driver only coordinates the commit.
Why: Each task writes its slice (Parquet/Delta files). The driver never materializes the full dataset. That is why write is safe and collect is not.
Common mistake: Using collect() 'to inspect' a 200 GB frame before writing.
Follow-up they will ask: What does show() send back?
beginnerWhere does a Python UDF actually run, and whose memory does it consume?
Driver vs executors · tap to open the answer
Say this first: In a separate python worker process on each executor node — not in the executor JVM heap and not on the driver.
Remember these
- UDF code runs in a python worker beside the executor JVM
- Rows cross a local socket: pickle per row, or Arrow per batch
- Plan shows BatchEvalPython or ArrowEvalPython
- Worker memory is outside the JVM heap; see spark.executor.pyspark.memory
Why: Your UDF body is pickled on the driver, shipped with the task, and executed by a python worker that the executor forks on the same machine. Rows travel from the JVM to that process and back over a local socket, serialised row by row for a plain udf and in Arrow batches for a pandas_udf. The plan node names the mode: BatchEvalPython for the row-at-a-time path, ArrowEvalPython for the vectorised one. Because the worker is its own process, its memory sits outside spark.executor.memory and is bounded by spark.executor.pyspark.memory, which is exactly why Python UDFs show up as container kills rather than heap errors. The driver's only involvement is serialising the closure, so a UDF that captures a large object makes every task payload large.
from pyspark.sql.functions import pandas_udf
@pandas_udf('double')
def to_usd(x):
return x * 1.09 # vectorised: ArrowEvalPython, not BatchEvalPythonCommon mistake: Saying a UDF 'runs on the driver' because it is defined in the notebook.
Follow-up they will ask: What does the plan lose the moment a Python UDF appears in it?
intermediateThe driver JVM has 8 GB and is barely half used, yet your notebook died with a Python memory error. Are those the same memory?
Driver vs executors · tap to open the answer
Say this first: No — the driver node runs a JVM heap and a separate Python process, and toPandas() fills the Python one.
Remember these
- spark.driver.memory is JVM heap only, not the Python process
- toPandas() holds a JVM copy and a Python copy at peak
- Arrow transfer is faster but does not remove the double footprint
- spark.driver.maxResultSize aborts before either process OOMs
Why: spark.driver.memory sizes the JVM heap where the SparkSession, Catalyst, and the scheduler live. Your PySpark code runs in a Python process on the same machine, and collect() or toPandas() moves rows from executors into the JVM first, then copies them across into Python. With Arrow enabled the transfer is columnar and faster, but it still means two live representations of the result at peak, so the node needs roughly double the result size free. Nothing in the JVM heap graph reflects the Python side, so the Executors tab can look calm while the kernel dies. On top of that, spark.driver.maxResultSize caps the total serialised task results an action may return and will abort the job before either process OOMs. Size the driver node's total RAM, not just the heap, whenever a pipeline pulls results into Python.
spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'true')
pdf = df.limit(50000).toPandas() # bounded, and the bound is the pointCommon mistake: Raising spark.driver.memory to fix a Python-side MemoryError on the driver node.
Follow-up they will ask: How would you return a 5 GB result to a Python client without this problem?
beginnerDefine job, stage, and task in one breath.
Jobs, stages, and tasks · tap to open the answer
Say this first: Job = one action. Stage = slice of the DAG between shuffles. Task = one partition in that stage.
Why: This is the vocabulary interviewers use to see if you've opened Spark UI. Mixing them is an instant no-hire for senior roles.
Common mistake: Calling an executor a stage.
Follow-up they will ask: How many tasks in a stage with 400 partitions?
intermediateA SQL query shows 3 jobs. Is that a bug?
Jobs, stages, and tasks · tap to open the answer
Say this first: Not necessarily — multiple actions, AQE, or Databricks SQL extra jobs.
Why: count + write is two jobs. AQE can add. Temporary views plus display add more. Map jobs to actions in the notebook.
Common mistake: Assuming one SQL string is always one job.
Follow-up they will ask: Where in Spark UI do you map SQL to jobs?
beginnerThe Spark UI marks a stage as 'skipped'. Did something go wrong?
Jobs, stages, and tasks · tap to open the answer
Say this first: No — skipped means its output was already available, usually from a previous shuffle or a cache, so Spark did not rerun it.
Remember these
- Skipped means output was reused, not that the stage failed
- It comes from a prior shuffle's map output or from a cache hit
- A skipped stage shows zero tasks and no errors
- Missing skips after caching means your cache is not being matched
Why: When a later job needs a stage whose shuffle output is still on the executors, the DAGScheduler reuses that output instead of recomputing it and shows the stage as skipped with zero tasks. The same happens when a cached RDD or DataFrame covers that part of the plan. It is a good sign: it means you are getting reuse out of work you already paid for, which is exactly what caching and a shared shuffle are supposed to buy. Failures look entirely different — a failed stage carries failed task counts and an error, and a stage that is genuinely not running at all simply never appears. If you expected skipped stages after caching and see full reruns instead, your cache is not matching the plan.
Common mistake: Reporting skipped stages as an incident because the word reads like something was dropped.
Follow-up they will ask: You cached a DataFrame and still see no skipped stages. Where do you look?
intermediateA stage has 200 tasks: 199 finish in 3 seconds, one takes 20 minutes. What is the stage duration, and what does that tell you about where to tune?
Jobs, stages, and tasks · tap to open the answer
Say this first: About 20 minutes — a stage ends when its slowest task ends, so the only thing worth tuning is that one task.
Remember these
- Stage duration is the max task duration, not the mean
- Summary Metrics max versus median is the skew measurement
- Idle cores mean more executors cannot help
- Fix the partition, not the cluster
Why: A stage is complete only when every one of its tasks is complete, and the next stage cannot start until then, so stage wall clock equals the maximum task duration, not the average. That is why averages are actively misleading here: the mean task duration in this stage is around nine seconds and tells you nothing about the twenty minutes you actually waited. The Summary Metrics table on the Stage page is built for this — compare the Max column against the Median column for Duration, Shuffle Read, and Spill, and the ratio names the problem. Adding executors does not help, because 199 cores are already idle waiting; the fix is whatever is making that one partition fat, which is usually key skew, a null or sentinel key, or an unsplittable input.
Common mistake: Quoting average task time in a performance review and concluding the stage is healthy.
Follow-up they will ask: Max and median are close but every task is slow. Now what?
beginnerWhat is the Spark DAG?
The Spark DAG · tap to open the answer
Say this first: The graph of RDD/DataFrame dependencies the scheduler uses to run stages.
Why: Narrow edges pipeline. Wide edges (shuffles) cut stages. Lineage lets Spark recompute lost partitions.
Common mistake: DAG as a Databricks workflow graph.
Follow-up they will ask: What happens to the DAG when an executor loses a cached partition?
intermediateWhy can a long lineage make a job fragile?
The Spark DAG · tap to open the answer
Say this first: Recompute after failure replays the whole chain; the DAG is huge.
Why: Checkpoint or write Delta to cut lineage. Spark UI DAG visualization gets unreadable after dozens of wide steps — that's a smell.
Common mistake: More cache() to 'shorten' a DAG without an action to materialize.
Follow-up they will ask: What's the difference between DAG Scheduler and Task Scheduler?
beginnerYour notebook cell chains five transformations and returns in 40 milliseconds. What did Spark build, and what did it not do?
The Spark DAG · tap to open the answer
Say this first: It extended a graph of operators in the driver and read nothing.
Remember these
- Transformations only extend the graph in the driver; no I/O happens
- Acyclic means a node never depends on its own output, so replay is safe
- The action submits the job — that is when the Jobs tab wakes up
- Column name errors still surface instantly, because analysis is eager
Why: Each transformation returns a new DataFrame that records its parent and the operation, so the cell only grows a graph in driver memory — no executor is contacted and no file is opened. The graph is directed because data flows one way, and acyclic because a node never depends on its own output, which is what makes replay after a failure safe. An action such as count() or write() is what hands the final logical plan to Catalyst, which produces a physical plan, after which the DAGScheduler cuts it into stages and submits tasks. That is why the Jobs tab stays empty until the action runs and then suddenly shows a job with several stages. The 40 ms was plan construction and analysis, including resolving your column names against the table schema.
df = spark.table("silver.orders").filter("amount > 0").select("id", "amount")
# nothing in the Jobs tab yet
df.count() # now a job appearsCommon mistake: Saying Spark 'reads the data lazily, row by row' — nothing is read at all until the action.
Follow-up they will ask: Which part of that 40 ms would still fail without a cluster?
intermediateThe DAG visualization shows a greyed-out stage labelled 'skipped'. Did Spark lose work?
The Spark DAG · tap to open the answer
Say this first: No — it reused shuffle files an earlier job already wrote, so that stage had nothing to recompute.
Remember these
- Skipped means the parent shuffle output still exists on disk
- Only shuffle boundaries can be skipped — narrow work is never reused
- Losing an executor can un-skip the stage and force a recompute
Why: Shuffle map output survives on the executors' local disks after the stage that wrote it finishes, and the driver keeps the map output locations. When a later job needs the same shuffle, the DAGScheduler sees the parent stage's output is already registered and marks it skipped instead of rerunning it. You get this whenever two actions share a prefix of the plan: the second job's early stages are grey and its Shuffle Read comes from existing files. It is the one form of reuse you get without calling cache(). The reuse disappears if an executor dies and no external shuffle service is holding those files, at which point the skipped stage is resubmitted.
agg = df.groupBy("region").count()
agg.count() # job 1 runs the shuffle
agg.show() # job 2: the map stage shows as skippedCommon mistake: Reading 'skipped' as data being dropped, or as a sign something failed.
Follow-up they will ask: How is that reuse different from cache()?
beginnerWhat is Catalyst?
Catalyst optimizer · tap to open the answer
Say this first: Spark SQL's optimizer: analysis → logical optimization → physical planning.
Why: It resolves names, pushes filters, reorders joins, then picks physical operators (broadcast vs SMJ). DataFrame/SQL get Catalyst; raw RDDs do not.
Common mistake: Catalyst as a storage format.
Follow-up they will ask: At which phase do unresolved attributes fail?
intermediateHow do you read explain('formatted') in an interview?
Catalyst optimizer · tap to open the answer
Say this first: Bottom is the scan. Look for PushedFilters, PartitionFilters, Exchange, and the join type.
Why: If the filter isn't in PushedFilters, you're scanning extra files. Exchange means shuffle. BroadcastHashJoin vs SortMergeJoin is the money line.
Common mistake: Pasting explain() without pointing at an operator.
Follow-up they will ask: What does AdaptiveSparkPlan mean?
beginnerYou misspell a column name and the job fails in under a second with no tasks launched. Which part of Catalyst caught it?
Catalyst optimizer · tap to open the answer
Say this first: The analyzer, which resolves names and types against the catalog before anything is optimized or executed.
Remember these
- Parse → analyze → optimize → physical plan → codegen
- Unresolved attributes fail in analysis, before any optimization
- An analysis failure costs no cluster time at all
Why: Your code first becomes an unresolved logical plan where column references are still strings. The analyzer looks each one up against the catalog's schema, binds data types, and resolves functions, so an unknown name dies here as an AnalysisException. Nothing has been optimized and no executor has been contacted, which is why it costs a second and no cluster time. Only after analysis succeeds does the optimizer apply its rewrite rules, and then the physical planner picks operators such as BroadcastHashJoin or SortMergeJoin. The practical lesson is to let Catalyst see the whole chain — breaking a pipeline into collected intermediate steps buys you the same error much later and much more expensively.
Common mistake: Expecting a typo to surface only when the action runs, because 'Spark is lazy'.
Follow-up they will ask: Which failures can analysis not catch?
intermediateCatalyst is described as both rule-based and cost-based. Which one actually does the work?
Catalyst optimizer · tap to open the answer
Say this first: Nearly all of it is rule-based rewriting; cost-based optimization is a narrower layer you have to feed with statistics.
Remember these
- Rule-based rewrites are unconditional and need no statistics
- CBO needs ANALYZE TABLE column stats to do anything useful
- AQE replaces estimates with measured shuffle sizes at runtime
Why: Logical optimization is a fixed set of rules applied until the tree stops changing — predicate pushdown, column pruning, constant folding, collapsing projections, null propagation. None of those need statistics, because they are always-true rewrites. Cost-based optimization, gated by spark.sql.cbo.enabled, uses table and column statistics to reorder joins and choose build sides, and it is meaningless without ANALYZE TABLE ... COMPUTE STATISTICS FOR COLUMNS. Without those statistics Spark falls back to size estimates derived from file sizes, which is why a filtered table is so often badly mis-estimated. AQE is the third layer and the one most teams actually rely on, because it works from measured shuffle sizes rather than estimates.
ANALYZE TABLE silver.orders COMPUTE STATISTICS FOR COLUMNS customer_id, order_date;Common mistake: Claiming Catalyst 'picks the cheapest plan by cost', as though a cost model drove every decision.
Follow-up they will ask: Why are file-size-based estimates wrong after a filter?
beginnerWhat is Tungsten?
Tungsten execution engine · tap to open the answer
Say this first: Spark's execution engine: binary rows, off-heap memory, whole-stage codegen.
Why: It avoids Java object overhead on the hot path. You get it with DataFrame/SQL. Python UDFs jump back to objects and kill the benefit.
Common mistake: Tungsten as a Databricks SKU.
Follow-up they will ask: What operator in explain() shows whole-stage codegen?
intermediateWhy does a Python UDF disable the fast path?
Tungsten execution engine · tap to open the answer
Say this first: Rows must be deserialized into Python, one call at a time (or batches for pandas UDFs).
Why: Plan shows BatchEvalPython / PythonUDF. Photon also bails out. Rewrite with Spark functions or Scala.
Common mistake: Pandas UDF as 'basically native'.
Follow-up they will ask: When is a pandas UDF acceptable?
intermediateIn explain("formatted") some operators carry a prefix like *(2). What is the star, and what does the number mean?
Tungsten execution engine · tap to open the answer
Say this first: The star marks operators compiled into whole-stage codegen; the number is the codegen stage they were fused into.
Remember these
- *(n) means this operator was fused into generated codegen stage n
- An operator with no star fell off the compiled fast path
- Exchange is never part of a codegen stage
Why: Whole-stage codegen takes a run of operators and generates a single Java method that loops over rows, instead of calling next() between operator objects. Everything sharing the same *(n) was fused into generated stage n, so the CPU runs one tight loop over Tungsten binary rows with no virtual dispatch and no intermediate objects. An operator printed without a star was not compiled — a Python UDF, an unsupported expression, or a fallback — and rows have to be materialized across that boundary. The number increments per codegen stage, so a plan with *(1) through *(5) has five fused regions, typically separated by Exchange nodes, which are never part of a codegen stage. Reading the stars is the fastest way to spot where the fast path broke.
df.groupBy("region").sum("amount").explain("formatted")
# *(1) HashAggregate and *(2) HashAggregate around the ExchangeCommon mistake: Reading the number as a stage of the job rather than a codegen stage of the plan.
Follow-up they will ask: Which operator in your last query lost its star?
beginnerAn executor has 32 GB. How much of that can you actually cache into?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Nowhere near all of it — the heap is split into reserved memory, a unified execution/storage region, and user memory.
Why: spark.memory.fraction decides how much of the usable heap the unified region gets, and inside that spark.memory.storageFraction is the share cache is guaranteed to hold onto. Whatever sits outside the fraction holds your own objects and Spark internals. Execution memory is what shuffles, sorts, joins, and aggregations consume.
Common mistake: Treating spark.executor.memory as the cache budget.
Follow-up they will ask: What happens when execution needs memory that cache is currently holding?
intermediateExecution and storage share one pool. Who is allowed to evict whom?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Execution can evict cached blocks; storage can never push out execution memory.
Why: Either side can borrow free space, but execution wins the tug-of-war and will evict cached partitions down to the storageFraction floor. That is why a DataFrame you cached quietly disappears and the next action rescans the source — the Storage tab shows the cached fraction dropping and the plan goes back to a FileScan.
Common mistake: Assuming cache() guarantees the data stays in memory for the rest of the session.
Follow-up they will ask: How would you spot that silent eviction in the UI?
beginnerA stage reports Spill (Memory) 40 GB and Spill (Disk) 6 GB. Why are the two numbers different?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: They measure the same records in two forms: deserialized in memory, then serialized and compressed on disk.
Remember these
- Spill (Memory) is deserialized size; Spill (Disk) is bytes written
- A large gap between them just means the data compresses well
- Any disk spill means execution memory was short for that task
Why: When execution memory fills, Spark serializes the in-memory records of a sort or aggregation and writes them to local disk as a sorted run. Spill (Memory) is how much that data occupied in deserialized form before the spill; Spill (Disk) is the bytes actually written after serialization and compression. A large ratio between them is normal and tells you the data compresses well, not that something is broken. What matters is that any disk spill at all means the task needed more execution memory than it was given, and merging those runs is extra I/O on the critical path. Compare the spilling task against the median task in the same stage before reaching for configuration.
Common mistake: Reading the two numbers as two separate spill events, or as evidence of a leak.
Follow-up they will ask: Is spill always worth eliminating?
intermediateBesides execution and storage there is user memory. What actually lives there, and when does it matter?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Everything Spark does not manage — your own objects, UDF state, closures, and class metadata.
Remember these
- User memory holds untracked objects created by your own code
- Spark never spills user memory — it can only run out
- High GC time with zero spill points at user memory, not execution
Why: Spark reserves a fixed slice of the heap, takes spark.memory.fraction of the remainder as the unified execution and storage pool, and leaves the rest as user memory. Nothing in that region is tracked or spilled by Spark, so a large lookup dictionary built inside mapPartitions, results accumulated in a local list, or a fat closure captured by a UDF all consume it silently. That is why lowering spark.memory.fraction is occasionally the right move: a job whose work lives in user-code data structures needs more unmanaged room, and raising the fraction starves it. The symptom is heap pressure and GC time with no spill recorded on the stage, because Spark genuinely does not know where the memory went.
spark.conf.set("spark.memory.fraction", "0.5") # more room for user-code objectsCommon mistake: Assuming every byte on the heap is inside the unified pool and therefore spillable.
Follow-up they will ask: How would you shrink a large per-partition lookup structure?
beginnerYour streaming query looks like ordinary DataFrame code. What is the engine actually doing under it?
Structured Streaming: an unbounded table · tap to open the answer
Say this first: Running a series of micro-batches: each batch reads a new offset range and executes the same incremental plan.
Why: The engine records which offsets it has processed in the checkpoint's offset log, plans a batch over the new range, and writes a commit when the batch finishes. That is why Spark UI shows a steady stream of small jobs rather than one long-running job.
Common mistake: Describing it as row-by-row streaming, like a message-queue consumer loop.
Follow-up they will ask: What guarantees the same row is not processed twice after a restart?
intermediateOn-call deleted the checkpoint directory to 'clear a stuck stream'. What did they just do?
Structured Streaming: an unbounded table · tap to open the answer
Say this first: Threw away the offset log, the commit log, and all state — the stream now restarts with no memory of what it processed.
Why: The checkpoint location holds offsets, commits, and the state store, and exactly-once depends on those lining up with the sink. Deleting it means reprocessing from the configured starting position — duplicates into an append sink, or silently skipped data if it starts at latest — and every aggregation's state is gone.
Common mistake: Treating the checkpoint directory as a cache you can safely clear when something looks stuck.
Follow-up they will ask: When is deleting a checkpoint actually the correct move?
intermediateYour windowed aggregation in append mode has written nothing for fifteen minutes, but input rows keep arriving. Is it broken?
Structured Streaming: an unbounded table · tap to open the answer
Say this first: Probably not — append emits a window only after the watermark passes its end, so output lags by the window plus the lateness threshold.
Remember these
- Append emits a window only after the watermark clears its end
- The watermark advances from event time in the data, not wall clock
- Update mode emits earlier but needs an upsert-capable sink
Why: In append mode a row must be final before it can be written, and a window is final only once the watermark has advanced past its end time. With a five-minute window and a ten-minute watermark, nothing can be emitted until event time has moved roughly fifteen minutes past a window's start, so a slow source looks like a stuck query. The watermark also advances from the maximum event time seen in the data, not from wall clock, so a source with no new events never advances it and never releases pending windows. Read the watermark and max event time from the query's lastProgress before assuming a failure. If you need visibility sooner, update mode emits changed windows every batch — at the cost of a sink that can handle upserts.
q = agg.writeStream.outputMode("append").start()
print(q.lastProgress["eventTime"]) # watermark and max event timeCommon mistake: Restarting the query because append mode 'produced no output', when it was waiting on the watermark by design.
Follow-up they will ask: What does complete mode do to state size?
beginnerWhat is data skew in Spark?
Data skew · tap to open the answer
Say this first: One key (or partition) has far more data than the others, so one task defines stage time.
Why: A shuffle sends every row with the same key to the same reducer, so the partition sizes follow the key distribution and nothing rebalances them. The classic offenders are a null country, a dominant 'US', or one enterprise customer_id that accounts for a third of the orders. A stage finishes when its slowest task finishes, so 199 tasks at 4 seconds and one at 9 minutes is a 9-minute stage no matter how many executors you add. In the Stages tab the giveaway is the task duration summary: compare the max against the median, and look for Spill (Disk) on that one task.
Common mistake: Skew as 'the cluster is unbalanced' without a key.
Follow-up they will ask: Which UI view shows skew in 10 seconds?
intermediateName three skew fixes and when each applies.
Data skew · tap to open the answer
Say this first: Filter/isolate hot keys, salt the key, AQE skew join.
Why: If nulls are junk, drop them. If one key is real (US), process it separately or salt. AQE split can help SMJ skew. Broadcast if the other side is small — skew on the fact may not matter.
Common mistake: repartition(1000) as a skew fix — the hot key still hashes to one partition.
Follow-up they will ask: Why doesn't more shuffle partitions fix a single hot key?
intermediateAQE skew join is enabled and the skewed stage is completely unchanged. Name two reasons.
Data skew · tap to open the answer
Say this first: The partition did not clear both skew thresholds, or the operator is not a shuffle join AQE can split.
Remember these
- Skew needs both the absolute byte threshold and the median factor
- Only shuffle joins are split — not aggregations, not broadcast joins
- A skew with no Exchange above it is invisible to AQE
Why: AQE calls a partition skewed only when it is both larger than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes and more than spark.sql.adaptive.skewJoin.skewedPartitionFactor times the median partition size. A partition that is huge in absolute terms but close to the median in a uniformly huge stage fails the second test, and a small-but-lopsided stage fails the first. The feature also applies only to joins executed as sort-merge or shuffled hash joins — it does not split a skewed groupBy, and it does not apply to a broadcast join because there is no shuffle partition to split. It needs spark.sql.adaptive.enabled on, and it works from a shuffle it can observe, so skew appearing before any Exchange is invisible to it. Check both thresholds against the actual partition sizes in the stage summary before concluding the feature is broken.
spark.conf.get("spark.sql.adaptive.skewJoin.skewedPartitionFactor")
spark.conf.get("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes")Common mistake: Enabling skewJoin and assuming any long task anywhere in the job will now be split.
Follow-up they will ask: How would you confirm from the UI that a split actually happened?
beginnerWhat is a driver OOM?
Driver OOM · tap to open the answer
Say this first: The driver JVM ran out of heap. Executors may still be healthy.
Why: Classic causes: collect, toPandas, createDataFrame from a huge list, broadcast of a large table. The notebook kernel dies.
Common mistake: Adding executors to fix a driver OOM.
Follow-up they will ask: Which Spark UI tab shows driver memory?
intermediateHow do you rewrite collect() on a 2 TB frame?
Driver OOM · tap to open the answer
Say this first: Don't. Aggregate, sample, write, or take(n).
Why: If you need a local ML sample, sample() then limit, or write a sampled Delta table and read elsewhere. Never collect the grain.
Common mistake: Increasing driver memory from 8 GB to 16 GB as the plan.
Follow-up they will ask: What's the Databricks-specific cousin of collect()?
beginnerOf show(20), count(), collect(), write(), toPandas() and take(5) — which ones pull rows to the driver?
Driver OOM · tap to open the answer
Say this first: collect(), toPandas(), take(5) and show(20) return rows to the driver; count() returns a number and write() returns nothing.
Remember these
- collect() and toPandas() return the whole dataset to one JVM
- take(n) and show(n) push a limit into the plan and stay bounded
- write() returns nothing — executors write the files directly
- Healthy executors plus a dead driver is the collect signature
Why: Every action returns its result to the driver, but the size of that result is what matters. collect() and toPandas() return the entire dataset, so the driver must hold all of it — and toPandas() briefly holds both the Spark rows and the pandas copy. take(n) and show(n) return at most n rows and Spark pushes the limit into the plan, so they are bounded by construction. count() returns a single number, and write() returns nothing because executors write the files themselves. The failure signature is unmistakable: the Executors tab shows no failed tasks and no ExecutorLostFailure, while the driver log has OutOfMemoryError or the notebook kernel simply disappears. Adding workers cannot help, because the workers were never the constraint.
df.count() # one number
df.take(5) # five rows
df.collect() # every row, in the driverCommon mistake: Believing count() is dangerous because it 'touches all the data' — the data is scanned on executors, not returned.
Follow-up they will ask: What makes toPandas() worse than collect() on the same frame?
intermediateA job fails with 'Total size of serialized results is bigger than spark.driver.maxResultSize'. What is that, and do you raise it?
Driver OOM · tap to open the answer
Say this first: A deliberate guard that aborts before task results fill the driver heap — raising it usually converts a clean error into an opaque heap OOM.
Remember these
- The cap is on total serialized task results per job, default 1 GB
- The error is a fast, legible failure instead of a driver heap OOM
- Setting it to 0 means unlimited, which only hides the next failure
Why: Task results travel back to the driver, and spark.driver.maxResultSize caps the total serialized size the driver will accept for a job, defaulting to 1 GB. Exceeding it means the job is trying to return far more data than a driver should hold, almost always a collect(), a toPandas(), or a very wide take(). The error is a feature: it fails fast with a clear message instead of letting the driver thrash and then die with Java heap space, which is much harder to diagnose. Raising the cap moves the failure, it does not remove it, and setting it to 0 for unlimited guarantees the next incident is an opaque driver OOM. Shrink the result instead — aggregate, limit, or write to a table and read it where you need it. The number is also a useful canary: if jobs routinely hit it, someone is using the driver as a data store.
spark.conf.get("spark.driver.maxResultSize") # raising this moves the failureCommon mistake: Setting spark.driver.maxResultSize to 0 and treating the resulting heap OOM as a different bug.
Follow-up they will ask: When is raising it actually defensible?
beginnerHow is executor OOM different from driver OOM?
Executor OOM · tap to open the answer
Say this first: A worker JVM dies. The driver usually stays up. You see ExecutorLostFailure / container killed.
Why: Causes: fat task (skew), huge partition, MEMORY_ONLY cache, big broadcast on the executor, explode() blowup.
Common mistake: Restarting the driver to fix an executor OOM.
Follow-up they will ask: What does Spark do after an executor is lost?
intermediateA task explodes a JSON array and dies. What's the fix?
Executor OOM · tap to open the answer
Say this first: The partition became huge after explode — not the scan.
Why: explode multiplies rows in one task. Repartition before explode, filter arrays, or process hot keys separately. Memory fraction / spark.memory.fraction is a last resort.
Common mistake: spark.executor.memory 64g as the first change.
Follow-up they will ask: How do you see the task that died?
intermediateYou cache a 200 GB table on a cluster with roughly 120 GB of storage memory. Nothing fails. What breaks later?
Executor OOM · tap to open the answer
Say this first: Only part of it is cached, and with MEMORY_ONLY the missing blocks are recomputed from source on every access.
Remember these
- A cache that does not fit is partial and silent — check Fraction Cached
- MEMORY_ONLY blocks that do not fit are recomputed, not spilled
- Cached blocks squeeze execution memory for every later stage
Why: Spark stores as many blocks as fit and then silently stops, so the Storage tab shows Fraction Cached below 100%. Every later action rescans the source for the missing blocks, so the query that was supposed to be fast is partly a full scan. Worse, the blocks that did fit occupy the unified pool, so the next shuffle has less execution memory, spills, and then evicts those cached blocks down to the storage floor anyway. The visible failure arrives later and somewhere else: an OOM or heavy spill in a join stage that has nothing to do with the cache. Check Fraction Cached before trusting a cache, prefer MEMORY_AND_DISK for anything expensive to rebuild, and unpersist what you are done with.
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count()
# Storage tab: Fraction Cached should read 100%Common mistake: Assuming cache() either works completely or raises an error.
Follow-up they will ask: Which storage level would you pick for an expensive-to-recompute frame?
beginnerHow does a broadcast join work?
Broadcast joins · tap to open the answer
Say this first: The driver (then each executor) gets a copy of the small table; the big table stays put.
Why: No shuffle of the fact table. Threshold is spark.sql.autoBroadcastJoinThreshold (default 10 MB, often raised). Too-large broadcast OOMs the driver or executors.
Common mistake: Broadcasting whichever table is mentioned first in SQL.
Follow-up they will ask: Which table should be broadcast?
intermediateBroadcast join OOM — driver or executor?
Broadcast joins · tap to open the answer
Say this first: Either: driver collects the build side; executors hold the hashed relation.
Why: Huge broadcast: driver collect of the dimension, then per-executor copy. Spark UI SQL shows broadcast exchange size. If the 'small' side is 8 GB, you chose wrong.
Common mistake: Raising executor memory without measuring broadcast size.
Follow-up they will ask: What's the hint syntax and when do you undo it?
beginnerYou join a 2 TB fact to a 5 MB lookup and the plan still shows SortMergeJoin. Name the likely reasons.
Broadcast joins · tap to open the answer
Say this first: The planner's size estimate is not 5 MB, automatic broadcast is disabled, or the join type cannot broadcast that side.
Remember these
- The planner uses an estimated size, which may not resemble reality
- autoBroadcastJoinThreshold set to -1 disables automatic broadcast
- Left outer can only broadcast the right side; full outer cannot broadcast
Why: Spark broadcasts automatically when its estimate for one side is under spark.sql.autoBroadcastJoinThreshold, and that estimate comes from table statistics or file sizes, not from the number in your head — a table with no statistics can be estimated as effectively unbounded. Setting the threshold to -1 disables automatic broadcast entirely, which some platforms do deliberately. The join type also constrains which side can be the build side: a left outer join can only broadcast its right side, and a full outer join cannot use a broadcast hash join at all. Finally the join must be an equi-join; a non-equality condition plans as a broadcast nested loop or a cartesian product instead. Check the threshold, run ANALYZE TABLE, and confirm the join type before reaching for a hint.
spark.conf.get("spark.sql.autoBroadcastJoinThreshold")
spark.sql("ANALYZE TABLE dim.lookup COMPUTE STATISTICS")Common mistake: Assuming a table that is small on disk is automatically small to the planner.
Follow-up they will ask: Which side gets broadcast in a right outer join?
intermediateWhy can a left outer join only broadcast its right side?
Broadcast joins · tap to open the answer
Say this first: The build side is the one looked up, and every left row must be preserved, which only works if the left side is streamed.
Remember these
- Build side is hashed and replicated; streamed side is iterated
- The outer side must be streamed, so it cannot be the build side
- A full outer join cannot use a broadcast hash join
Why: A broadcast hash join builds a hash table from one side and streams the other through it, emitting matches per streamed row. A left outer join must emit every left row, matched or not, so the left side has to be the streamed side and the right has to be the build side. Reverse it and a left row with no match could sit on an executor that never sees it, so the semantics break. The logic inverts for a right outer join, where only the left side can be broadcast, and a full outer join needs both sides preserved, so it cannot use a broadcast hash join at all and plans as a sort-merge join. This is why a broadcast hint on the wrong side is silently ignored: Spark is not disobeying you, the strategy is invalid for that join type.
fact.join(broadcast(dim), "dim_id", "left") # valid
dim.join(broadcast(fact), "dim_id", "left") # hint ignoredCommon mistake: Reporting that 'the broadcast hint did nothing' without checking the join type.
Follow-up they will ask: What does Spark do with a left anti join on a small right side?
beginnerWhat is Adaptive Query Execution?
Adaptive Query Execution · tap to open the answer
Say this first: Spark re-plans later stages at runtime using real sizes.
Why: Three headlines: coalesce shuffle partitions, switch join strategy, handle skewed joins. Enabled by default in modern Spark / Databricks.
Common mistake: AQE as a replacement for designing a good join key.
Follow-up they will ask: Does AQE run before the first stage?
intermediateWhen will AQE not save you?
Adaptive Query Execution · tap to open the answer
Say this first: A single skewed key, a Python UDF, or a broadcast that's already too big.
Why: Coalesce won't split a hot key. Join conversion needs a truly small side. Skew join has limits. You still need a sane plan and files.
Common mistake: Turning every knobs to true and calling it architecture.
Follow-up they will ask: Which AQE feature shows up as extra stages in the UI?
beginnerWhy does the plan in the SQL / DataFrame tab look different from what df.explain() printed in your notebook?
Adaptive Query Execution · tap to open the answer
Say this first: explain() shows the pre-execution plan; the UI shows the final plan after AQE re-planned it with real statistics.
Remember these
- explain() before execution reports isFinalPlan=false
- AQE re-plans at shuffle boundaries using measured statistics
- The SQL / DataFrame tab holds the plan that actually ran
Why: With AQE on, the physical plan is wrapped in an AdaptiveSparkPlan node, and before execution it reports isFinalPlan=false — what you printed is Catalyst's estimate-based guess. As each shuffle map stage completes, AQE reads its real output statistics and may coalesce shuffle partitions, convert a join to a broadcast, or split a skewed partition, then re-plans the remainder of the query. The SQL / DataFrame tab shows that final plan with isFinalPlan=true and marks the nodes AQE changed. So a plan pasted from a notebook is a hypothesis about what will run, and the UI is the record of what did run. Debug against the UI plan — chasing a SortMergeJoin that AQE already converted wastes an afternoon.
df.explain("formatted") # AdaptiveSparkPlan isFinalPlan=false
df.collect() # then read the SQL / DataFrame tabCommon mistake: Debugging the notebook plan and never opening the SQL tab to see what AQE did.
Follow-up they will ask: Which part of a query can AQE never change?
intermediateWhere does AQE get its runtime statistics, and what follows from that?
Adaptive Query Execution · tap to open the answer
Say this first: From the map output statistics of completed shuffle stages, which is why it can only act at Exchange boundaries.
Remember these
- Statistics come from completed shuffle map stages, per output partition
- No shuffle in the query means no AQE decision point
- The first shuffle still runs with the static partition count
Why: When a shuffle map stage finishes it reports the size of every output partition back to the driver. AQE uses those numbers — actual bytes per reducer partition, not estimates — to decide whether to coalesce partitions, convert a join, or split a skew. Two consequences follow. A query with no shuffle gets essentially nothing from AQE, because there is no measurement point. And the very first shuffle is still planned from estimates and still launches spark.sql.shuffle.partitions tasks, so AQE cannot fix the partitioning of the stage it is measuring, only of the stages after it. That is why people see 200 tasks in the first stage and a handful in the second and conclude AQE is inconsistent. The measurement point is the Exchange.
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.get("spark.sql.shuffle.partitions") # still the initial countCommon mistake: Expecting AQE to fix the partitioning of the first shuffle, which it must run before it can measure.
Follow-up they will ask: What does spark.sql.adaptive.coalescePartitions.initialPartitionNum change?
Practice by topic
- Spark Overview
- Spark Architecture
- Driver & Executors
- Execution Flow
- DataFrame
- Spark SQL
- Transformations
- Actions
- Lazy Evaluation
- RDD
- Partitions
- Repartition vs Coalesce
- Shuffle
- Joins
- Caching
- File Formats
- Window Functions
- UDFs
- Job / Stage / Task
- DAG
- Catalyst
- Tungsten
- Memory Management
- Structured Streaming
- Broadcast Join
- Data Skew
- AQE
- Driver OOM
- Executor OOM
