DataYatriSpark, seen under the hood
Simulate
spark/internals
beginner
Connecting…

Driver vs executors

The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.

Lesson 3 of 29 · Spark path

Explain it at my level

  1. Driver
  2. Executors
  3. Result
Watch the canvas:client / drivercluster managerCatalyst / shuffleexecutorsLive simulation
Driver JVMSparkSession · schedulerExecutor 1tasks doneExecutor 2tasks doneExecutor 3tasks doneTwo kinds of JVM. Debug the one that actually died.
Driver = coordinator JVM. Executors = data JVMs. collect() copies the cluster onto the coordinator.