Interview/Spark

Spark interview questions: Cache and persist

Interactive Spark interview questions on Cache and persist. Practice with the matching lesson. Caching stores computed partitions so later actions do not recompute the DAG. Unpersist when you are done.

Lesson · Simulation

What does cache() actually do?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Cache and persist

MEMORY_ONLY cache, then executor OOM

Symptoms

  • Storage tab full
  • One executor lost
  • Driver still up

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What does cache() actually do?

Cache and persist · tap to open the answer

Say this first: Marks the DataFrame to be stored after the next action. It is lazy.

Why: First action computes and stores partitions (MEMORY_AND_DISK by default for DataFrames). Second action can skip recomputation if the plan matches and data still fits.

Common mistake: cache() as an action that runs immediately.

Follow-up they will ask: How do you uncache?

Lesson · Simulation

intermediate

When is cache the wrong answer?

Cache and persist · tap to open the answer

Say this first: When you read the data once, or when the cached set is bigger than memory and thrashes disk.

Why: Cache for reuse in the same session (ML iterative, branching QA). For a single write, cache adds memory pressure. Checkpoint if lineage is huge.

Common mistake: Caching every intermediate table in a 30-step ETL.

Follow-up they will ask: MEMORY_ONLY vs MEMORY_AND_DISK — which fails harder?

Lesson · Simulation

senior

Cached DataFrame, then an executor dies. What happens?

Cache and persist · tap to open the answer

Say this first: Lost partitions recompute from lineage — or fail if checkpoint wasn't used and shuffle files are gone.

Why: Cache is not a durable store. Storage tab shows cached partitions per executor. Executor loss → recompute. That's why checkpoint or a Delta write is the durable form.

Senior depth: Storage tab + stage that skips the scan. If the scan is back, cache missed or was evicted.

Common mistake: Treating cache as a table.

Follow-up they will ask: How do you see cache hit vs recompute in the UI?

Lesson · Simulation

beginner

You call df.cache() and the next cell is just as slow as before. What was missed?

Cache and persist · tap to open the answer

Say this first: cache() only marks the plan; nothing is stored until an action materializes the partitions.

Remember these

  • cache() is lazy; the first action pays and populates
  • count() materializes every partition, show() may populate only some
  • The Storage tab is empty until materialization happens
  • persist with no subsequent action does nothing at all

Why: cache() and persist() register the DataFrame's analyzed plan as an InMemoryRelation and return immediately — no data moves. The first action after that computes the plan and fills the cache as a side effect, so that action pays full price and only later ones can benefit. Which action you use matters too: count() touches every partition and therefore populates all of them, while show() or limit(10) may only compute enough partitions to satisfy the limit, leaving the cache partially filled. The Storage tab is the proof — a cached entry appears only once something materialized it, and Fraction Cached tells you how much of it landed. A persist with no following action is simply a no-op that cost you a line of code.

gold.cache()
gold.count()        # this action fills the cache
gold.groupBy("region").count().show()   # this one reads it

Common mistake: Describing cache() as an action that moves data at the moment it is called.

Follow-up they will ask: Does unpersist() also wait for an action?

Lesson · Simulation

intermediate

The Storage tab shows 'Fraction Cached 63%' for your DataFrame. What does the next action actually do?

Cache and persist · tap to open the answer

Say this first: It reads the 63% from the block manager and recomputes the other 37% from lineage.

Remember these

  • Caching is per-partition; missing partitions recompute from lineage
  • MEMORY_ONLY drops partitions that do not fit; MEMORY_AND_DISK spills them
  • Eviction is LRU across everything cached in the session
  • Select and filter before caching to shrink what you are asking to store

Why: Caching is per-partition, not all-or-nothing. A partition is stored only if it fits where the storage level says it should go, so with MEMORY_ONLY any partition that did not fit is simply dropped, and with MEMORY_AND_DISK it spills to the executor's local disk instead of being lost. Either way the next action reads the cached partitions from the block manager and re-executes the full lineage for the missing ones, which is why a partly cached frame can be barely faster than an uncached one. A fraction below 100% usually means the cached set is bigger than the storage memory available, or that other cached frames in the same session evicted yours under LRU. The remedies are to cache less — project columns away and filter before caching — to switch to MEMORY_AND_DISK, or to unpersist frames you no longer need.

from pyspark import StorageLevel
narrow = gold.select("customer_id", "region", "amount")
narrow.persist(StorageLevel.MEMORY_AND_DISK)
narrow.count()

Common mistake: Reading a partial fraction as 'cache is working' and never checking why 37% is missing.

Follow-up they will ask: Two notebooks on the same cluster both cache large frames. What happens?

Lesson · Simulation

advanced

You cached a DataFrame, then wrote a new query with an extra filter. Does it hit the cache?

Cache and persist · tap to open the answer

Say this first: Only if the new query's plan contains the cached plan as a subtree — Spark matches on the analyzed logical plan, not on the table name.

Remember these

  • Cache matching is by analyzed logical plan subtree, not by name
  • InMemoryTableScan in the plan proves the hit; FileScan proves the miss
  • Build from the cached DataFrame object to guarantee the match
  • A cache hit forfeits predicate pushdown into the file format

Why: When you cache, Spark records the analyzed logical plan of that DataFrame in the cache manager. On every subsequent query it tries to substitute an InMemoryRelation wherever it finds that exact plan as a subtree. Building on the cached DataFrame — cached_df.filter(...) — matches, so the plan shows InMemoryTableScan with the filter applied above it. Re-reading the source and writing what looks like the same transformations usually does not match, because the plans must be equivalent after analysis and small differences in projection order, aliases, or added columns are enough to miss. The way to confirm is explain('formatted'): InMemoryTableScan means a hit, a FileScan back on the source means a miss. Also note what you give up on a hit — the filter now runs over cached rows instead of being pushed into the Parquet scan, so a highly selective predicate can be faster uncached.

gold.cache(); gold.count()
gold.where("region = 'EU'").explain("formatted")   # look for InMemoryTableScan

Senior depth: Partly. The in-memory columnar format stores columns separately, so an InMemoryTableScan reads only the requested ones and can skip batches using per-batch statistics. What you cannot recover is the pruning you gave up at cache time: whatever columns were in the DataFrame when you cached it are what got materialized. Cache a narrow projection, not the whole table, and that problem disappears.

Common mistake: Assuming any query that mentions the same table reuses the cache.

Follow-up they will ask: Does a cached DataFrame still get column pruning when you select two of fifty columns?

Lesson · Simulation

senior

A 40-iteration algorithm gets slower every iteration even though you cache after each step. What is growing?

Cache and persist · tap to open the answer

Say this first: The logical plan — caching stores the data but does not truncate lineage, so planning cost grows with every iteration.

Remember these

  • cache stores data but keeps lineage; the plan keeps growing
  • Symptom is rising driver time between stages, not slower tasks
  • checkpoint writes to reliable storage and truncates the lineage
  • localCheckpoint truncates using executor disk and gives up recoverability

Why: Each iteration stacks another layer of transformations onto the previous plan, and cache does not cut that chain: it adds an InMemoryRelation node but keeps the full lineage underneath so lost partitions can be recomputed. By iteration 30 the driver is analyzing and optimizing a plan with thousands of nodes before any task is submitted, so you see driver CPU rising and the gap between cells growing while executor time stays flat. checkpoint() is the operator that actually cuts it: it writes the DataFrame to the directory set by sparkContext.setCheckpointDir and replaces the lineage with a read of that file, so the next iteration starts from a one-node plan. localCheckpoint() does the same truncation using executor disk instead of reliable storage, which is faster but loses the data — and therefore the ability to recover — if an executor dies. A good pattern for long iterative jobs is to cache every iteration and checkpoint every few.

spark.sparkContext.setCheckpointDir("/mnt/chk/iterative")
for i in range(40):
    df = step(df).cache()
    if i % 5 == 0:
        df = df.checkpoint()

Senior depth: Compare the wall clock of the cell against the summed stage durations in the Spark UI for the jobs it submitted. If the cell took ninety seconds and the stages account for twelve, the remaining time is driver-side analysis, optimization, and code generation. A driver thread dump during the gap will show the optimizer rules on the stack.

Common mistake: Adding executors when the time is being spent on the driver building a plan.

Follow-up they will ask: How would you prove the time is in planning rather than execution?

Lesson · Simulation

senior

Give me three concrete ways adding cache() can make a job slower.

Cache and persist · tap to open the answer

Say this first: It costs a materialization pass, it steals memory from execution, and it blocks pushdown into the source format.

Remember these

  • Materializing the cache is a cost paid even if reuse never happens
  • Storage memory and execution memory compete, so caching can cause shuffle spill
  • Reading from cache gives up predicate and column pushdown into Parquet
  • Cache narrow and filtered frames with real reuse, then unpersist

Why: First, populating the cache is real work: the partitions must be built and written into the block manager, and with a serialized storage level they are serialized and compressed too. If the frame is read once, that is pure overhead. Second, cached blocks live in storage memory in the same unified region as execution memory, so a heavy shuffle in the same job now has less room and starts spilling — the slowdown appears as Spill (Disk) in a stage that has nothing to do with the cache. Third, once a query reads from InMemoryTableScan, there is no Parquet scan left to push a filter into, so a selective predicate that previously skipped most row groups now scans every cached row. The rule is to cache a narrow, filtered frame that two or more downstream branches genuinely reuse, and to unpersist it as soon as the reuse is finished.

candidates = events.where("event_date >= '2026-10-01'").select("user_id", "event_type")
candidates.cache(); candidates.count()
# ... two branches read it ...
candidates.unpersist()

Senior depth: When the reuse spans jobs, sessions, or teams, or when the frame is large enough that holding it in memory squeezes execution. A Delta table is durable, readable by other jobs, carries statistics for pruning, and survives executor loss — all things a cache does not do. Cache is a within-session optimization; a table is a contract.

Common mistake: Caching every intermediate 'to be safe' and then blaming the shuffle spill it caused on executor memory.

Follow-up they will ask: When would you write an intermediate table instead of caching at all?

Lesson · Simulation