DataYatriSpark, seen under the hood
Simulate

Simulation/Spark

Educational simulation
Spark

Driver OOM

collect() piles rows onto the driver. Executors stay green. The notebook dies.

Watch the canvas:driver JVMexecutorcollect() rowsLive simulation
Driver JVMSparkSession · schedulerdriver heap0.8g / 8gExecutor 1tasks doneExecutor 2tasks doneExecutor 3tasks doneexec heap2.1g / 16gexec heap2.1g / 16gexec heap2.1g / 16gcollect() / toPandas() copy every row onto spark.driver.memoryExecutors tab green. Driver stderr: Java heap space. Do not add workers.show() / limit() / write() — the table never enters the driver JVM
Driver OOM is a data-path bug. The workers already did their job.