DataYatriSpark, seen under the hood
Simulate
spark/core
beginner
Connecting…

Spark SQL: the same engine, a different keyboard

SQL strings and the DataFrame API compile to the same plan. Temp views, the catalog, and when names actually resolve.

Lesson 6 of 29 · Spark path

Explain it at my level

  1. SQL / DataFrame
  2. Parse
  3. Analyze
  4. Optimize
  5. Physical plan
Watch the canvas:DataFrame APISQL textcataloglogical planphysical planactionLive simulation
Two front doorsDataFrame APIdf.filter(col("region") == "US")SQL textSELECT * FROM orders WHERE region='US'Catalogname → plantemp view / Unity / HiveParseUnresolved logical plannot built yetAnalyzeAnalyzerlocked🔒OptimizeCatalyst optimizerlocked🔒ExecutePhysical planlocked🔒collect() — Job 0no action yet🔒A temp view is a name, not a copy. Session-scoped, zero bytes.
spark.sql() and df.filter() parse to the same tree, resolve against the same catalog, and end as one identical physical plan.