DataYatriSpark, seen under the hood
Simulate

Simulation/Spark

Educational simulation
Spark

Cluster Architecture

Driver, cluster manager, and executors — who does what.

Watch the canvas:drivercluster managerexecutorLive simulation
DriverSparkSession · Catalyst · DAGCluster managerYARN · Kubernetes · DatabricksExecutor 1worker-014 cores · 8 GBExecutor 2worker-024 cores · 8 GBExecutor 3worker-034 cores · 8 GBrowsshow() → few rows to driver write() → storage, driver stays small
Three roles, one rule: big data stays on executors.