DataYatriSpark, seen under the hood
Simulate
spark/internals
advanced
Connecting…

Executor memory: execution wins, cache loses

Execution and storage share one pool. Execution can evict your cache, then spill to disk, then die.

Lesson 23 of 29 · Spark path

Explain it at my level

  1. Regions
  2. Split
  3. Eviction
  4. One-way
  5. Spill
Watch the canvas:reserveduser memoryexecutionstorage / cachespill to diskLive simulation
Executor JVM · spark.executor.memory = 8gcontainer = heap + spark.executor.memoryOverheadHeap regions · spark.memory.fraction = 0.6Resv300 MB — fixed, never allocatableUseryour objects, UDF state, class metadataUnified0.6 × (heap − 300 MB) — one shared poolBoundary · spark.memory.storageFraction = 0.5 (a start line, not a wall)Execution evicts Storagea task needs buffer space and takes itMEMORY_ONLY blocks are dropped outrightStorage never evicts Executionevicting a buffer would fail the taskeviction is one-directional — cache losesSpill (Memory) / Spill (Disk)sorted runs written locally, then mergedThen OutOfMemoryErrorGC thrash → ExecutorLostFailure → stage retry8g configured is not 8g usable. 300 MB off the top, then × 0.6.
One executor heap: reserved, user, and a unified pool whose Execution/Storage boundary slides one way only — toward Execution.