spark/fundamentals
beginner
Connecting…

Spark cluster architecture

Driver, cluster manager, and executors. How a SparkSession becomes running JVMs.

Explain it at my level

  1. SparkSession
  2. Cluster manager
  3. Executors
  4. Storage
Driver · SparkSession · DAG scheduler