Cache and persist
MEMORY_ONLY cache, then executor OOM
Symptoms
- Storage tab full
- One executor lost
- Driver still up
Cache and persist interview questions
Interactive Spark interview questions on Cache and persist. Same topic as /learn/spark/caching. Caching stores computed partitions so later actions do not recompute the DAG. Unpersist when you are done.
Question 1 of 3
beginner
Cache and persistWhat does cache() actually do?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat does cache() actually do?
Cache and persist · tap to open the answer
Short: Marks the DataFrame to be stored after the next action. It is lazy.
Detailed: First action computes and stores partitions (MEMORY_AND_DISK by default for DataFrames). Second action can skip recomputation if the plan matches and data still fits.
Common mistake: cache() as an action that runs immediately.
Follow-up: How do you uncache?
intermediateWhen is cache the wrong answer?
Cache and persist · tap to open the answer
Short: When you read the data once, or when the cached set is bigger than memory and thrashes disk.
Detailed: Cache for reuse in the same session (ML iterative, branching QA). For a single write, cache adds memory pressure. Checkpoint if lineage is huge.
Common mistake: Caching every intermediate table in a 30-step ETL.
Follow-up: MEMORY_ONLY vs MEMORY_AND_DISK — which fails harder?
seniorCached DataFrame, then an executor dies. What happens?
Cache and persist · tap to open the answer
Short: Lost partitions recompute from lineage — or fail if checkpoint wasn't used and shuffle files are gone.
Detailed: Cache is not a durable store. Storage tab shows cached partitions per executor. Executor loss → recompute. That's why checkpoint or a Delta write is the durable form.
Senior: Storage tab + stage that skips the scan. If the scan is back, cache missed or was evicted.
Common mistake: Treating cache as a table.
Follow-up: How do you see cache hit vs recompute in the UI?