DataYatriSpark, seen under the hood
Simulate
spark/fundamentals
beginner
Connecting…

Spark cluster architecture

Driver, cluster manager, and executors. How a SparkSession becomes running JVMs.

Lesson 2 of 29 · Spark path

Explain it at my level

  1. Client
  2. Driver
  3. Catalyst
  4. DAG
  5. Executors
  6. Result
Watch the canvas:client / drivercluster managerCatalyst / shuffleexecutorsLive simulation
Clientspark-submit job.pyCluster managerYARN · K8s · DatabricksDriver JVMSparkSession · DAG schedulerInside the driverJob → Stages → Taskswaiting for a physical planCatalyst optimizerquery planning — no cluster I/O yetUnresolvedAnalyzedOptimizedPhysicalCompute layerExecutor 1waiting for tasks4 cores · 8 GBExecutor 2waiting for tasks4 cores · 8 GBExecutor 3waiting for tasks4 cores · 8 GBrowsClient → cluster manager. The driver does not exist yet.
From spark-submit to show(): driver plans, manager allocates, executors compute.