spark/core
beginner
Connecting…Partitions: the unit of parallelism
One partition is one chunk of data. One task processes one partition in a stage. Size them for 128–256 MB.
Explain it at my level
- Files
- Partitions
- Tasks
Step 1 of 4
orders table
orders · ~1 TB across parquet files