spark/internals
beginner
Connecting…

Driver vs executors

The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.

Explain it at my level

  1. Driver
  2. Executors
  3. Result
Driver · SparkSession · DAG scheduler