DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Spark interview questions: Catalyst optimizer

Interactive Spark interview questions on Catalyst optimizer. Practice with the matching lesson. Catalyst rewrites your query four times. Exchange in the physical plan is the shuffle you will pay for.

Lesson · Simulation

beginner
Catalyst optimizer

What is Catalyst?

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Catalyst optimizer

Filter on date still scans the whole lake

Symptoms

  • Table not partitioned by date
  • Or filter wrapped in a UDF
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

What is Catalyst?

Catalyst optimizer · tap to open the answer

Short: Spark SQL's optimizer: analysis → logical optimization → physical planning.

Detailed: It resolves names, pushes filters, reorders joins, then picks physical operators (broadcast vs SMJ). DataFrame/SQL get Catalyst; raw RDDs do not.

Common mistake: Catalyst as a storage format.

Follow-up: At which phase do unresolved attributes fail?

Lesson · Simulation

intermediate

How do you read explain('formatted') in an interview?

Catalyst optimizer · tap to open the answer

Short: Bottom is the scan. Look for PushedFilters, PartitionFilters, Exchange, and the join type.

Detailed: If the filter isn't in PushedFilters, you're scanning extra files. Exchange means shuffle. BroadcastHashJoin vs SortMergeJoin is the money line.

Common mistake: Pasting explain() without pointing at an operator.

Follow-up: What does AdaptiveSparkPlan mean?

Lesson · Simulation

senior

Catalyst picked SMJ. Stats were wrong. What's your move?

Catalyst optimizer · tap to open the answer

Short: ANALYZE TABLE / fix stats, add a broadcast hint if you know the size, or enable AQE join conversion.

Detailed: Don't hint forever as a substitute for stats. Hints rot. Measure with actual size after filters — AQE sees runtime sizes.

Senior: When you've measured the build side after filters and documented why stats cannot see it (UDF, JDBC, etc.).

Common mistake: Broadcast hint on a table that will grow 20×.

Follow-up: When is a hint the right senior answer?

Lesson · Simulation