DataYatriSpark, seen under the hood
Simulate
spark/fundamentals
beginner
Connecting…

Why Spark is lazy

Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.

Lesson 9 of 29 · Spark path

Explain it at my level

  1. Read
  2. Transform
  3. Action
Watch the canvas:plan node (nothing runs)action (job starts)Live simulation
read parquetFileScan nodefilter regionFilter nodegroupBy regionAggregate nodeshow()action — job submittedDriver-side plan only. Cluster CPU used so far: none.Job 0 submitted. Now — and only now — executors open files.Because the filter existed before the scan, Parquet skips row groups.
Three of these four lines cost nothing. Only the last one starts a cluster.