DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Spark interview questions: Cache and persist

Interactive Spark interview questions on Cache and persist. Practice with the matching lesson. Caching stores computed partitions so later actions do not recompute the DAG. Unpersist when you are done.

Lesson · Simulation

beginner
Cache and persist

What does cache() actually do?

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Cache and persist

MEMORY_ONLY cache, then executor OOM

Symptoms

  • Storage tab full
  • One executor lost
  • Driver still up
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

What does cache() actually do?

Cache and persist · tap to open the answer

Short: Marks the DataFrame to be stored after the next action. It is lazy.

Detailed: First action computes and stores partitions (MEMORY_AND_DISK by default for DataFrames). Second action can skip recomputation if the plan matches and data still fits.

Common mistake: cache() as an action that runs immediately.

Follow-up: How do you uncache?

Lesson · Simulation

intermediate

When is cache the wrong answer?

Cache and persist · tap to open the answer

Short: When you read the data once, or when the cached set is bigger than memory and thrashes disk.

Detailed: Cache for reuse in the same session (ML iterative, branching QA). For a single write, cache adds memory pressure. Checkpoint if lineage is huge.

Common mistake: Caching every intermediate table in a 30-step ETL.

Follow-up: MEMORY_ONLY vs MEMORY_AND_DISK — which fails harder?

Lesson · Simulation

senior

Cached DataFrame, then an executor dies. What happens?

Cache and persist · tap to open the answer

Short: Lost partitions recompute from lineage — or fail if checkpoint wasn't used and shuffle files are gone.

Detailed: Cache is not a durable store. Storage tab shows cached partitions per executor. Executor loss → recompute. That's why checkpoint or a Delta write is the durable form.

Senior: Storage tab + stage that skips the scan. If the scan is back, cache missed or was evicted.

Common mistake: Treating cache as a table.

Follow-up: How do you see cache hit vs recompute in the UI?

Lesson · Simulation