Spark beginner interview questions

Short: Code → logical plan → job → stages → tasks → (optional shuffle) → result.

Detailed: Transformations are lazy. The action builds a plan, splits it at shuffles into stages, runs one task per partition, then returns a small result or writes files.

Common mistake: Saying 'Spark runs the DataFrame line by line'.

Follow-up: Where does a shuffle sit in that chain?

Lesson · Simulation

Short: Each wide transformation (shuffle) is a stage boundary.

Detailed: Narrow steps (filter, map, broadcast join) stay in one stage. groupBy, join-without-broadcast, repartition cut a new stage because data must move.

Common mistake: Counting jobs instead of stages when debugging runtime.

Follow-up: How do you see stage boundaries in the physical plan?

Lesson · Simulation

Short: On an action — not when you write filter or groupBy.

Detailed: The lazy plan is a recipe. show(), count(), write(), collect() cook it. Until then Spark has not scanned storage.

Common mistake: Wrapping every line in count() 'to be safe'.

Follow-up: What does an unresolved logical plan still not know?

Lesson · Simulation

Short: The optimizer is allowed to push predicates and prune partitions.

Detailed: Your code order is not the physical order. Predicate pushdown and partition pruning happen in analysis/optimization. That is why a late filter can still skip files.

Common mistake: Assuming Python line order is the execution order.

Follow-up: How do you prove a filter was pushed into the scan?

Lesson · Simulation

Short: The work triggered by one action.

Detailed: Each show/count/write/collect is a job (sometimes more with AQE). A job is not a Databricks Workflow. A job contains stages.

Common mistake: Calling the whole notebook 'one Spark job'.

Follow-up: How many jobs does df.cache(); df.count(); df.count() typically create?

Lesson · Simulation

Short: AQE can coalesce, switch join strategy, and split skew after it sees sizes.

Detailed: The first stages run, statistics update, later stages change. Spark UI shows SQL / AQE details. That is expected, not a second user action.

Common mistake: Thinking extra stages mean the user clicked twice.

Follow-up: Which AQE feature changes a shuffle join into a broadcast?

Lesson · Simulation

Short: A shuffle — a wide dependency.

Detailed: Narrow transformations pipeline inside a stage. When keys must co-locate (groupBy, sort merge join, repartition), Spark inserts an Exchange and a new stage.

Common mistake: Saying each transformation is its own stage.

Follow-up: Is a filter a new stage?

Lesson · Simulation

Short: The slowest task — the straggler.

Detailed: Stages wait for all tasks. Median is a vanity metric. Read max task duration, spill, and shuffle read on that task.

Common mistake: Optimizing average task time while one key holds the stage.

Follow-up: Which UI chart shows this immediately?

Lesson · Simulation

Short: A partition is a slice of data. A task is the unit of work that reads that slice in a stage.

Detailed: One partition → one task in that stage. Executors run many tasks over time. Do not call an executor a task.

Common mistake: Using partition, task, and executor as synonyms.

Follow-up: If you have 200 partitions and 10 executors, how many tasks in a narrow stage?

Lesson · Simulation

Short: Task scheduling overhead dwarfs the work. Each task has launch cost.

Detailed: Tiny files / too many partitions → thousands of 20 ms tasks. Spark UI shows high scheduler delay. Coalesce or compact files.

Common mistake: repartition(10000) 'for more parallelism' on a 200 MB table.

Follow-up: How do you pick a partition count for a 200 GB shuffle?

Lesson · Simulation

Short: Only what the action asked for — a few show() rows, a count, or nothing for write.

Detailed: Shuffle moves data between executors. It does not dump the table on the driver. write() commits files; the driver stays small. collect() is the action that pulls.

Common mistake: Believing every shuffle returns the full dataset to the notebook.

Follow-up: Does saveAsTable send rows through the driver?

Lesson · Simulation

Short: Map tasks write shuffle files locally; reducers fetch over the network; spill hits disk again if a reducer partition is fat.

Detailed: Spark UI: shuffle write, shuffle read, fetch wait, spill memory/disk. A healthy shuffle is balanced. A sick one has one reducer reading 10× bytes.

Common mistake: Only looking at CPU when a stage is an Exchange.

Follow-up: What is shuffle partition size aiming at?

Lesson · Simulation

Short: A distributed engine that processes large datasets in parallel across a cluster.

Detailed: Spark splits data into partitions, runs a task per partition on executors, and only ships results when an action runs. It is not a database and it does not store your lake.

Common mistake: Calling Spark a 'database' or saying it always holds data in memory.

Follow-up: What is lazy about a DataFrame transformation?

Lesson · Simulation

Short: Transformations build a plan. Actions execute it.

Detailed: groupBy / filter / join are lazy. Spark waits for show, count, write, collect. That is when jobs, stages, and tasks appear in Spark UI.

Common mistake: Thinking Spark is slow at parse time because the notebook looks busy.

Follow-up: Name three actions that trigger a job.

Lesson · Simulation

Short: Driver coordinates. Executors compute. Cluster manager places the JVMs.

Detailed: The driver builds the DAG and tracks tasks. Executors run tasks and hold cache. The cluster manager (YARN, K8s, Databricks) allocates machines. Big data stays on executors and storage — not in the driver heap.

Common mistake: Saying the driver 'processes the data'.

Follow-up: Where does collect() put the result?

Lesson · Simulation

Short: The driver owns the SparkSession, DAG, and task scheduler. Executors are workers, not the brain.

Detailed: Lose the driver JVM and there is no one to retry tasks or return the action result. Databricks job clusters fail the run; notebooks disconnect.

Common mistake: Restarting one executor to 'fix a driver OOM'.

Follow-up: What Spark UI page is served from the driver?

Lesson · Simulation

Short: The original distributed collection. DataFrames compile down toward RDD/Tungsten execution.

Detailed: You rarely write RDD code now. Interviewers want: partitions, lineage, and that DataFrames are the API you should use.

Common mistake: Starting a new ETL in RDD map/reduce.

Follow-up: What is lineage on an RDD?

Lesson · Simulation

Short: Catalyst optimizes the plan; Tungsten uses off-heap/binary rows. RDDs are Java objects.

Detailed: RDD map on Row objects boxes everything. DataFrame expressions can whole-stage codegen. You also get predicate pushdown on files.

Common mistake: rdd.filter because 'it's more control'.

Follow-up: When is an RDD the right tool?

Lesson · Simulation

Short: A distributed table with a schema and a lazy query plan.

Detailed: Rows are partitioned across executors. Operations return a new plan, not a local pandas object. spark.createDataFrame on a huge Python list still starts on the driver.

Common mistake: Treating it like pandas (df[i] loops, collect to 'feel the data').

Follow-up: How do you print the schema without executing a job?

Lesson · Simulation

Short: Almost never for ETL. DataFrames get Catalyst and Tungsten. RDDs skip that.

Detailed: Use RDDs only for custom partitioning or APIs that still need them. groupBy/join/filter belong on DataFrames/Datasets.

Common mistake: rdd.map for a column expression Catalyst could optimize.

Follow-up: What do you lose when you call rdd.map on a DataFrame?

Lesson · Simulation

Short: Transformation: new DataFrame, lazy. Action: job, side effect or result.

Detailed: filter, select, groupBy, join are transformations. show, count, write, collect, take are actions. groupBy alone does not shuffle until an action.

Common mistake: Saying groupBy 'runs the shuffle immediately'.

Follow-up: Is cache a transformation or an action?

Lesson · Simulation

Short: Narrow: filter/map — no shuffle. Wide: groupBy/join — shuffle and a new stage.

Detailed: Narrow tasks read only their partition. Wide tasks wait on Exchange. Broadcast join is wide in API but not a shuffle of the fact table.

Common mistake: Calling join always a shuffle join.

Follow-up: Why can a filter after a groupBy not reduce shuffle bytes of that groupBy?

Lesson · Simulation

Short: count → a number. show → a few printed rows. collect → all rows. write → nothing (files on storage).

Detailed: take/limit also pull a small result. foreach runs on executors. The dangerous one is collect on a large frame.

Common mistake: Using collect() as the default way to 'see data'.

Follow-up: Which action is safest to check a pipeline ran?

Lesson · Simulation

Short: It still executes the full plan, including shuffles.

Detailed: count after a wide transformation pays the shuffle. It is a cheap result, not a cheap job. Sometimes Spark can optimize count on a scan; not after a join.

Common mistake: Sprinkling count() after every transform in production.

Follow-up: When is approx_count_distinct the better interview answer?

Lesson · Simulation

Short: Spark records the recipe and waits for an action.

Detailed: That lets Catalyst optimize the whole chain (predicate pushdown, join reorder) instead of running each line. It also means errors can appear late.

Common mistake: Evaluating each transformation when the line runs, like pandas.

Follow-up: How do you force execution without collect()?

Lesson · Simulation

Short: The cached plan is the unfiltered one; the new plan does not match, or you never materialized the cache.

Detailed: cache() is lazy. Need an action to populate Storage. A new filter is a different lineage unless you cache after the filter.

Common mistake: Assuming persist survives arbitrary extra operators.

Follow-up: How do you confirm a scan hit cache in Spark UI?

Lesson · Simulation

Short: A chunk of a DataFrame that one task reads in a stage.

Detailed: Input partitions often follow files. After a shuffle, spark.sql.shuffle.partitions (or AQE) decides how many. One partition = one task, not one executor.

Common mistake: One partition per executor as a rule.

Follow-up: Who decides input partitions for a Parquet scan?

Lesson · Simulation

Short: repartition shuffles to a new count. coalesce reduces without a full shuffle (can stay unbalanced).

Detailed: repartition(n) for even parallelism or a join key. coalesce(n) after a filter that dropped most rows, to avoid a shuffle. Don't coalesce to 1 on a huge frame except for a tiny output.

Common mistake: coalesce(1) to 'make a single CSV' on a 400 GB table.

Follow-up: What does partitionBy on write control versus RDD partitions?

Lesson · Simulation

Short: Moves records so equal keys land on the same reducer. Disk + network + a stage boundary.

Detailed: Map tasks write shuffle files. Reducers fetch their slice. groupBy, distinct, join (non-broadcast), and repartition all shuffle.

Common mistake: Thinking shuffle is 'Spark being slow' rather than a specific Exchange.

Follow-up: Which Spark UI numbers prove a shuffle?

Lesson · Simulation

Short: Network fetch, disk spill, and waiting on the slowest reducer.

Detailed: Shuffle write/read bytes, fetch wait, spill. A skewed key makes one reducer read most of the data. Compression and AQE help only if the plan is sane.

Common mistake: Adding CPU cores when fetch wait is the bottleneck.

Follow-up: What is a shuffle block?

Lesson · Simulation

Short: Broadcast if one side fits in memory on every executor. Shuffle (sort-merge) if both sides are large.

Detailed: Broadcast replicates the small table. SMJ partitions both by key and sorts. A wrong broadcast of a 'small' 10 GB table OOMs executors.

Common mistake: Broadcasting the fact table because 'joins should be broadcast'.

Follow-up: How do you force a broadcast in Spark SQL?

Lesson · Simulation

Short: Statistics said both sides were large, or AQE/broadcast threshold was below the build side.

Detailed: Check table stats, spark.sql.autoBroadcastJoinThreshold, and filters that Catalyst didn't push (so size is wrong). AQE can switch later if enabled.

Common mistake: Hints without checking stats.

Follow-up: What happens if stats are stale after a 10× load?

Lesson · Simulation

Short: Marks the DataFrame to be stored after the next action. It is lazy.

Detailed: First action computes and stores partitions (MEMORY_AND_DISK by default for DataFrames). Second action can skip recomputation if the plan matches and data still fits.

Common mistake: cache() as an action that runs immediately.

Follow-up: How do you uncache?

Lesson · Simulation

Short: When you read the data once, or when the cached set is bigger than memory and thrashes disk.

Detailed: Cache for reuse in the same session (ML iterative, branching QA). For a single write, cache adds memory pressure. Checkpoint if lineage is huge.

Common mistake: Caching every intermediate table in a 30-step ETL.

Follow-up: MEMORY_ONLY vs MEMORY_AND_DISK — which fails harder?

Lesson · Simulation

Short: Driver: SparkSession, Catalyst, DAG, task scheduling. Executor: tasks, cache, shuffle files.

Detailed: Your notebook code until an action is driver-side. After the action, work is split into tasks that executors run on partitions. Results come back only if the action asks (show, collect).

Common mistake: Thinking each executor has its own SparkSession you should create.

Follow-up: Why is creating a SparkSession inside a foreach a bug?

Lesson · Simulation

Short: Executors write partitions in parallel to storage. The driver only coordinates the commit.

Detailed: Each task writes its slice (Parquet/Delta files). The driver never materializes the full dataset. That is why write is safe and collect is not.

Common mistake: Using collect() 'to inspect' a 200 GB frame before writing.

Follow-up: What does show() send back?

Lesson · Simulation

Short: Job = one action. Stage = slice of the DAG between shuffles. Task = one partition in that stage.

Detailed: This is the vocabulary interviewers use to see if you've opened Spark UI. Mixing them is an instant no-hire for senior roles.

Common mistake: Calling an executor a stage.

Follow-up: How many tasks in a stage with 400 partitions?

Lesson · Simulation

Short: Not necessarily — multiple actions, AQE, or Databricks SQL extra jobs.

Detailed: count + write is two jobs. AQE can add. Temporary views plus display add more. Map jobs to actions in the notebook.

Common mistake: Assuming one SQL string is always one job.

Follow-up: Where in Spark UI do you map SQL to jobs?

Lesson · Simulation

Short: The graph of RDD/DataFrame dependencies the scheduler uses to run stages.

Detailed: Narrow edges pipeline. Wide edges (shuffles) cut stages. Lineage lets Spark recompute lost partitions.

Common mistake: DAG as a Databricks workflow graph.

Follow-up: What happens to the DAG when an executor loses a cached partition?

Lesson · Simulation

Short: Recompute after failure replays the whole chain; the DAG is huge.

Detailed: Checkpoint or write Delta to cut lineage. Spark UI DAG visualization gets unreadable after dozens of wide steps — that's a smell.

Common mistake: More cache() to 'shorten' a DAG without an action to materialize.

Follow-up: What's the difference between DAG Scheduler and Task Scheduler?

Lesson · Simulation

Short: Spark SQL's optimizer: analysis → logical optimization → physical planning.

Detailed: It resolves names, pushes filters, reorders joins, then picks physical operators (broadcast vs SMJ). DataFrame/SQL get Catalyst; raw RDDs do not.

Common mistake: Catalyst as a storage format.

Follow-up: At which phase do unresolved attributes fail?

Lesson · Simulation

Short: Bottom is the scan. Look for PushedFilters, PartitionFilters, Exchange, and the join type.

Detailed: If the filter isn't in PushedFilters, you're scanning extra files. Exchange means shuffle. BroadcastHashJoin vs SortMergeJoin is the money line.

Common mistake: Pasting explain() without pointing at an operator.

Follow-up: What does AdaptiveSparkPlan mean?

Lesson · Simulation

Short: Spark's execution engine: binary rows, off-heap memory, whole-stage codegen.

Detailed: It avoids Java object overhead on the hot path. You get it with DataFrame/SQL. Python UDFs jump back to objects and kill the benefit.

Common mistake: Tungsten as a Databricks SKU.

Follow-up: What operator in explain() shows whole-stage codegen?

Lesson · Simulation

Short: Rows must be deserialized into Python, one call at a time (or batches for pandas UDFs).

Detailed: Plan shows BatchEvalPython / PythonUDF. Photon also bails out. Rewrite with Spark functions or Scala.

Common mistake: Pandas UDF as 'basically native'.

Follow-up: When is a pandas UDF acceptable?

Lesson · Simulation

Short: One key (or partition) has far more data than the others, so one task defines stage time.

Detailed: Classic: null country, 'US', or one customer_id. Median task is fine; max task is the SLA.

Common mistake: Skew as 'the cluster is unbalanced' without a key.

Follow-up: Which UI view shows skew in 10 seconds?

Lesson · Simulation

Short: Filter/isolate hot keys, salt the key, AQE skew join.

Detailed: If nulls are junk, drop them. If one key is real (US), process it separately or salt. AQE split can help SMJ skew. Broadcast if the other side is small — skew on the fact may not matter.

Common mistake: repartition(1000) as a skew fix — the hot key still hashes to one partition.

Follow-up: Why doesn't more shuffle partitions fix a single hot key?

Lesson · Simulation

Short: The driver JVM ran out of heap. Executors may still be healthy.

Detailed: Classic causes: collect, toPandas, createDataFrame from a huge list, broadcast of a large table. The notebook kernel dies.

Common mistake: Adding executors to fix a driver OOM.

Follow-up: Which Spark UI tab shows driver memory?

Lesson · Simulation

Short: Don't. Aggregate, sample, write, or take(n).

Detailed: If you need a local ML sample, sample() then limit, or write a sampled Delta table and read elsewhere. Never collect the grain.

Common mistake: Increasing driver memory from 8 GB to 16 GB as the plan.

Follow-up: What's the Databricks-specific cousin of collect()?

Lesson · Simulation

Short: A worker JVM dies. The driver usually stays up. You see ExecutorLostFailure / container killed.

Detailed: Causes: fat task (skew), huge partition, MEMORY_ONLY cache, big broadcast on the executor, explode() blowup.

Common mistake: Restarting the driver to fix an executor OOM.

Follow-up: What does Spark do after an executor is lost?

Lesson · Simulation

Short: The partition became huge after explode — not the scan.

Detailed: explode multiplies rows in one task. Repartition before explode, filter arrays, or process hot keys separately. Memory fraction / spark.memory.fraction is a last resort.

Common mistake: spark.executor.memory 64g as the first change.

Follow-up: How do you see the task that died?

Lesson · Simulation

Short: The driver (then each executor) gets a copy of the small table; the big table stays put.

Detailed: No shuffle of the fact table. Threshold is spark.sql.autoBroadcastJoinThreshold (default 10 MB, often raised). Too-large broadcast OOMs the driver or executors.

Common mistake: Broadcasting whichever table is mentioned first in SQL.

Follow-up: Which table should be broadcast?

Lesson · Simulation

Short: Either: driver collects the build side; executors hold the hashed relation.

Detailed: Huge broadcast: driver collect of the dimension, then per-executor copy. Spark UI SQL shows broadcast exchange size. If the 'small' side is 8 GB, you chose wrong.

Common mistake: Raising executor memory without measuring broadcast size.

Follow-up: What's the hint syntax and when do you undo it?

Lesson · Simulation

Short: Spark re-plans later stages at runtime using real sizes.

Detailed: Three headlines: coalesce shuffle partitions, switch join strategy, handle skewed joins. Enabled by default in modern Spark / Databricks.

Common mistake: AQE as a replacement for designing a good join key.

Follow-up: Does AQE run before the first stage?

Lesson · Simulation

Short: A single skewed key, a Python UDF, or a broadcast that's already too big.

Detailed: Coalesce won't split a hot key. Join conversion needs a truly small side. Skew join has limits. You still need a sane plan and files.

Common mistake: Turning every knobs to true and calling it architecture.

Follow-up: Which AQE feature shows up as extra stages in the UI?

Lesson · Simulation