Interview/Spark

Spark interview questions: Why Spark is lazy

Interactive Spark interview questions on Why Spark is lazy. Practice with the matching lesson. Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.

Lesson · Simulation

What does lazy evaluation mean for a DataFrame?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Why Spark is lazy

Pipeline 'green' in interactive, red in jobs

Symptoms

  • Notebook had extra cells that never ran in the job
  • Job starts at the action

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What does lazy evaluation mean for a DataFrame?

Why Spark is lazy · tap to open the answer

Say this first: Spark records the recipe and waits for an action.

Why: That lets Catalyst optimize the whole chain (predicate pushdown, join reorder) instead of running each line. It also means errors can appear late.

Common mistake: Evaluating each transformation when the line runs, like pandas.

Follow-up they will ask: How do you force execution without collect()?

Lesson · Simulation

intermediate

You persist a DataFrame, then add a filter, then count. Why is cache unused?

Why Spark is lazy · tap to open the answer

Say this first: The cached plan is the unfiltered one; the new plan does not match, or you never materialized the cache.

Why: cache() is lazy. Need an action to populate Storage. A new filter is a different lineage unless you cache after the filter.

Common mistake: Assuming persist survives arbitrary extra operators.

Follow-up they will ask: How do you confirm a scan hit cache in Spark UI?

Lesson · Simulation

senior

When is laziness a production footgun?

Why Spark is lazy · tap to open the answer

Say this first: When invalid tables, bad schemas, or huge plans only fail at the action — in a job cluster at 2am.

Why: Validate schema and table existence early. Unit-test explain() on representative data. Don't leave 15 debug actions that each rerun a shuffle.

Common mistake: Adding count() everywhere as 'validation'.

Follow-up they will ask: What is a cheap eager check that isn't a full shuffle?

Lesson · Simulation

beginner

Which of these does work right now: df.printSchema(), df.explain(), df.limit(5), df.count()?

Why Spark is lazy · tap to open the answer

Say this first: printSchema and explain are driver-only metadata operations, limit is another lazy transformation, and only count submits a job.

Remember these

  • printSchema reads a resolved plan; no stage is submitted
  • explain does driver-side planning work but submits no job
  • limit() is a transformation producing LocalLimit and GlobalLimit
  • count() is the only action here and runs the whole plan

Why: printSchema reads the already-resolved schema off the logical plan, so it costs nothing beyond what resolution already did at read time. explain runs Catalyst through analysis and planning and prints the result, which is real driver work but submits no stage — it is the tool to reach for when you want to inspect a plan without paying for it. limit(5) is a transformation: it appends a LocalLimit and GlobalLimit pair and returns a new DataFrame, which is why df.limit(5) on its own prints nothing and shows nothing in the Jobs tab. count() is the only action in the list, and it executes the entire plan including any shuffle, then returns a single number. Knowing which of the four is free is what lets you debug a plan on a production-sized table without launching anything.

df.printSchema()      # free
df.explain('formatted')  # driver only
df.limit(5)           # still lazy
df.count()            # Job submitted

Common mistake: Thinking limit(5) is an action because show(5) looks similar and does run.

Follow-up they will ask: Which one would you use to check a filter pushed down, on a 10 TB table?

Lesson · Simulation

intermediate

You called cache() then count(). The Storage tab says the DataFrame is 55% cached. What happens on the next query?

Why Spark is lazy · tap to open the answer

Say this first: Cached blocks are served from memory or local disk; the uncached partitions are recomputed from the source every time you read.

Remember these

  • DataFrame cache() is MEMORY_AND_DISK, so overflow spills rather than drops
  • Memory-only levels recompute uncached partitions from lineage
  • Storage tab Fraction Cached and Size on Disk distinguish the two
  • Cache competes with execution memory and can cause shuffle spill

Why: cache() on a DataFrame uses the MEMORY_AND_DISK storage level, so when the storage pool fills, blocks are written to the executor's local disk rather than dropped — a partially cached relation usually means memory pressure pushed part of it to disk, and reads of those blocks pay disk I/O instead of a recompute. With an explicitly memory-only level the behaviour is harsher: blocks that do not fit are simply not cached, and every read recomputes them from lineage, which on an expensive upstream plan can make the cached version slower than no cache at all. Either way the Storage tab's Fraction Cached and Size on Disk columns tell you which situation you are in, and the plan shows InMemoryTableScan with the InMemoryRelation underneath. Caching also competes for the same unified pool as execution memory, bounded below by spark.memory.storageFraction, so a large cache can push your shuffles into spilling. If the relation does not fit, the right move is usually to cache a narrower projection or a filtered subset, or to write an intermediate table instead.

df.select('id', 'amount').filter('amount > 0').cache().count()
# cache a narrow, filtered projection rather than the whole table

Common mistake: Caching a wide table and counting the cache as free, while the shuffle in the same job starts spilling.

Follow-up they will ask: How would you decide between caching this and writing an intermediate table?

Lesson · Simulation

advanced

When is breaking lineage with checkpoint() the right answer instead of cache()?

Why Spark is lazy · tap to open the answer

Say this first: When the plan itself is the problem — an iterative loop whose lineage keeps growing — because checkpoint truncates the plan while cache only stores results under it.

Remember these

  • cache() keeps the lineage; checkpoint() replaces it with a file read
  • Growing plans in iterative loops are a planning problem, not a data problem
  • localCheckpoint() truncates lineage but is not fault tolerant
  • Writing an intermediate Delta table is often the better checkpoint

Why: cache() stores the computed partitions but leaves the full lineage attached, so the plan the driver carries keeps growing and any evicted block is recomputed through that entire history. checkpoint() writes the dataset to the directory set by sparkContext.setCheckpointDir and replaces the plan with a plain read of those files, so the lineage is genuinely cut and recovery reads from reliable storage. That matters for loops: a feature-engineering or graph-iteration job that appends operators every pass eventually spends more time in Catalyst than in execution, and no amount of caching fixes a plan that is thousands of nodes deep. localCheckpoint() truncates the lineage without the durable write, which is faster and loses the data if an executor is lost — fine inside a single job, wrong as a recovery point. Both are actions in their eager form, so they cost a full execution at the point you call them. In a lake pipeline, writing an intermediate Delta table is often the better version of the same idea, because it is inspectable, reusable by the next run, and visible in table history.

spark.sparkContext.setCheckpointDir('/tmp/ckpt')
for i in range(50):
    df = df.withColumn(f'f{i}', expr('amount * 1.01'))
    if i % 10 == 0:
        df = df.checkpoint()

Senior depth: The question I ask is what exactly is expensive: the bytes or the tree. Cache addresses bytes, checkpoint addresses the tree. Measuring the gap between query submission and the first task launch tells you which one you have, and that measurement takes ten seconds while guessing wrong costs a day.

Common mistake: Reaching for cache() against a driver that is slow at planning, when the problem is plan depth rather than recomputation.

Follow-up they will ask: How would you tell planning time from execution time in the UI?

Lesson · Simulation

senior

You want the pipeline to fail fast on a bad schema at 02:00 instead of ninety minutes in. What cheap eager checks do you add?

Why Spark is lazy · tap to open the answer

Say this first: Metadata-only assertions up front: resolve every source table, compare schemas, probe one row per input, and assert the plan's pushdown before the heavy work starts.

Remember these

  • Resolving a table's schema is a catalog call, not a job
  • limit(1).count() per source costs one task and catches empty inputs
  • Assert PartitionFilters in the plan to catch pruning regressions
  • Delta constraints enforce data rules on every write, not just this job

Why: Lazy evaluation means a missing table or a renamed column only surfaces when an action reaches that part of the plan, so a validation block at the top of the job converts a late failure into an immediate one. Resolving spark.table(name).schema for every input costs a catalog round trip and no stage, and comparing it against an expected StructType catches renames, type drift, and added columns before any data moves. A bounded existence probe — limit(1).count() or isEmpty() on each source — costs one task per input and catches empty or unreadable partitions. Calling explain('formatted') and asserting that PartitionFilters is non-empty on the big fact scan catches the day someone wraps the partition column in a cast and turns a pruned read into a full scan. For data-level guarantees, let the storage layer carry them: Delta NOT NULL and CHECK constraints are validated on every write by every writer, which is stronger than any assertion in one job. What you should not add is a count() per transformation, because that is a full re-execution dressed up as a check.

expected = spark.table('contracts.orders_v3').schema
actual = spark.table('bronze.orders').schema
assert actual == expected, f'schema drift: {actual}'

Senior depth: Separate the two failure classes. Contract failures — missing table, wrong schema, lost pruning — are cheap to detect and belong at the top of the job. Data failures belong in table constraints and expectations so they are enforced for every writer. Mixing them produces jobs full of count() calls that cost real money and still miss the thing that broke.

Common mistake: Calling it validation while sprinkling count() between transformations, which re-runs the plan every time.

Follow-up they will ask: Which of these checks belongs in the pipeline and which belongs in the table definition?

Lesson · Simulation

architect

A team wants to pass a DataFrame from one Workflow task to the next instead of writing a table. Explain why that cannot work.

Why Spark is lazy · tap to open the answer

Say this first: A DataFrame is a handle into one live SparkSession; a separate task is a different session and usually a different JVM, so there is nothing to hand over.

Remember these

  • A DataFrame references a live session and plan; it cannot be serialised
  • Each task gets its own SparkSession and its own session catalog
  • The durable handoff is a table name plus a version, not an object
  • Snapshot identifiers at task boundaries make partial reruns deterministic

Why: The object holds a reference to a session, a plan tree, and driver-side state such as cached relations — none of which is serialisable or meaningful outside the process that built it. Even where two tasks share a cluster, each gets its own SparkSession with its own session catalog, so temp views and caches from the first are invisible to the second. The only durable handoff is something both sessions can resolve independently: a table name, a path, or a table name plus a version. Delta makes that handoff precise, because the producing task can emit the committed version and the consumer can read exactly that snapshot, which also makes a partial rerun deterministic. Treat each task boundary as a contract with a schema and a snapshot identifier, and the pipeline becomes restartable at any step instead of all-or-nothing. If the intermediate genuinely must not be durable, the honest answer is that it belongs inside one task, not split across two.

# task 1
dbutils.jobs.taskValues.set('version', version)
# task 2
v = dbutils.jobs.taskValues.get('etl', 'version')
spark.read.option('versionAsOf', v).table('silver.orders')

Senior depth: Task boundaries are where restartability is won or lost. Durable intermediates with explicit versions cost storage and buy you the ability to rerun step four at 03:00 without rerunning steps one through three. That trade is almost always worth it, and the teams that resist it are usually optimising a cost they have never measured.

Common mistake: Trying to share state across tasks with a global temp view and assuming it survives because the cluster is the same.

Follow-up they will ask: What should the task boundary contract contain besides the table name?

Lesson · Simulation