DataYatriSpark, seen under the hood
Simulate

Simulation/Databricks

Educational simulation
Databricks

OPTIMIZE & Clustering

Tiny files compact, then Z-ORDER / CLUSTER BY skip filters.

Watch the canvas:tiny filecompactedclustered by regionLive simulation
2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB2 MB16 files × 2 MB = 16 tiny tasks. Compact before you tune Spark.
Layout is a performance feature. OPTIMIZE and clustering change how files sit.