Catalyst optimizer

Catalyst rewrites your query four times. Exchange in the physical plan is the shuffle you will pay for.

Lesson 21 of 29 · Spark path

Explain it at my level

Watch the canvas:logical stagephysical planLive simulation

Catalyst is four trees. Each one is a strictly better description of the same query.