DataYatriSpark, seen under the hood
Simulate
spark/fundamentals
beginner
Connecting…

What is Apache Spark?

Spark is a distributed compute engine. You write a plan; a driver turns it into tasks that executors run on partitions.

Lesson 1 of 29 · Spark path

Explain it at my level

  1. Driver
  2. Cluster
  3. Executors
  4. Result
Watch the canvas:client / drivercluster managerCatalyst / shuffleexecutorsLive simulation
Your coderead.parquet · show()Driverplans · does not scan the lakeCluster managerYARN · K8s · DatabricksExecutor 1idleExecutor 2idleExecutor 3idlerowsNothing ran. Spark is waiting so it can optimize.
Spark is a scheduler plus a query optimizer. You talk to a driver; executors hold the data.