DataYatriSpark, seen under the hood
Simulate
spark/core
beginner
Connecting…

Partitions: the unit of parallelism

One partition is one chunk of data. One task processes one partition in a stage. Size them for 128–256 MB.

Lesson 11 of 29 · Spark path

Explain it at my level

  1. Files
  2. Partitions
  3. Tasks
Watch the canvas:partition (data)task (work)Live simulation
orders~1 TB across Parquet files on S3P0250 MBTask 0reads P0P1240 MBTask 1reads P1P2260 MBTask 2reads P2P3250 MBTask 3reads P310,000 files of 2 MB = 10,000 tasks that finish in 40 ms. Compact first.
Partition and task are the same unit seen from opposite sides.