collect() and toPandas() pull the cluster into one JVM. Executors stay green. The driver is the process that dies.
Explain it at my level
Drag to rotate · scroll to zoom · tank height = heap size
Spark 0.4 GB / 4.6 GB · user 0.5 GB / 3.1 GB
Spark 0.8 GB / 9.4 GB · user 0.5 GB / 6.3 GB
Spark 0.8 GB / 9.4 GB · user 0.5 GB / 6.3 GB
Spark 0.8 GB / 9.4 GB · user 0.5 GB / 6.3 GB
Spark memory layout — why the exception fires
executor Spark memory = (16g − 300MB) × spark.memory.fraction 0.6 = 9.4 GB
driver Spark memory = (8g − 300MB) × 0.6 = 4.6 GB · maxResultSize = 1.0 GB
Driver heap 8.0 GB is not all yours. collect()/toPandas() still have to fit deserialized rows in that JVM. Executor Spark memory is 9.4 GB each and does not take this result.
Code
rows = spark.table("gold.orders").collect()Spark UI
Executors tab green. Driver stderr: Java heap space. No ExecutorLostFailure.
Fix
show() / limit() / write() — do not assemble the fact table on the driver.