DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Adaptive Query Execution interview questions

Interactive Spark interview questions on Adaptive Query Execution. Same topic as /learn/spark/aqe. AQE uses runtime stats to coalesce shuffle partitions, change join strategy, and split skewed partitions.

Lesson · Simulation

beginner
Adaptive Query Execution

What is Adaptive Query Execution?

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Adaptive Query Execution

AQE broadcast conversion OOMs a growing dim

Symptoms

  • Worked last quarter
  • Dim crossed the threshold at runtime
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

What is Adaptive Query Execution?

Adaptive Query Execution · tap to open the answer

Short: Spark re-plans later stages at runtime using real sizes.

Detailed: Three headlines: coalesce shuffle partitions, switch join strategy, handle skewed joins. Enabled by default in modern Spark / Databricks.

Common mistake: AQE as a replacement for designing a good join key.

Follow-up: Does AQE run before the first stage?

Lesson · Simulation

intermediate

When will AQE not save you?

Adaptive Query Execution · tap to open the answer

Short: A single skewed key, a Python UDF, or a broadcast that's already too big.

Detailed: Coalesce won't split a hot key. Join conversion needs a truly small side. Skew join has limits. You still need a sane plan and files.

Common mistake: Turning every knobs to true and calling it architecture.

Follow-up: Which AQE feature shows up as extra stages in the UI?

Lesson · Simulation

senior

AQE coalesced 2000 shuffle partitions down to 12, and now one task is huge. What happened?

Adaptive Query Execution · tap to open the answer

Short: Coalesce packed leftover data into too few partitions, or skew was hidden.

Detailed: Coalesce is for empty/small partitions. If data is uneven, you can get fat tasks. Inspect post-AQE partition sizes. Disable coalesce for that query or salt.

Senior: spark.sql.adaptive.enabled and compare SQL UI plans — same query, different later stages.

Common mistake: Blaming AQE without looking at the adaptive plan.

Follow-up: How do you show AQE on/off in an interview demo?

Lesson · Simulation