DataYatriSpark, seen under the hood
Simulate

Simulation/Spark

Educational simulation
Spark

Cluster Architecture

Driver, cluster manager, and executors — who does what.

Watch the canvas:client / drivercluster managerCatalyst / shuffleexecutorsLive simulation
Clientspark-submit job.pyCluster managerYARN · K8s · DatabricksDriver JVMSparkSession · DAG schedulerInside the driverJob → Stages → Taskswaiting for a physical planCatalyst optimizerquery planning — no cluster I/O yetUnresolvedAnalyzedOptimizedPhysicalCompute layerExecutor 1waiting for tasks4 cores · 8 GBExecutor 2waiting for tasks4 cores · 8 GBExecutor 3waiting for tasks4 cores · 8 GBrowsClient → cluster manager. The driver does not exist yet.
From spark-submit to show(): driver plans, manager allocates, executors compute.