Tungsten execution engine
Same SQL 5× slower after a 'tiny' UDF
Symptoms
- One email-normalize Python UDF
- Rest of the query native
Tungsten execution engine interview questions
Interactive Spark interview questions on Tungsten execution engine. Same topic as /learn/spark/tungsten. Tungsten stores rows off-heap and generates JVM bytecode for whole stages so Spark SQL is not a naive iterator of JVM objects.
Question 1 of 3
beginner
Tungsten execution engineWhat is Tungsten?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is Tungsten?
Tungsten execution engine · tap to open the answer
Short: Spark's execution engine: binary rows, off-heap memory, whole-stage codegen.
Detailed: It avoids Java object overhead on the hot path. You get it with DataFrame/SQL. Python UDFs jump back to objects and kill the benefit.
Common mistake: Tungsten as a Databricks SKU.
Follow-up: What operator in explain() shows whole-stage codegen?
intermediateWhy does a Python UDF disable the fast path?
Tungsten execution engine · tap to open the answer
Short: Rows must be deserialized into Python, one call at a time (or batches for pandas UDFs).
Detailed: Plan shows BatchEvalPython / PythonUDF. Photon also bails out. Rewrite with Spark functions or Scala.
Common mistake: Pandas UDF as 'basically native'.
Follow-up: When is a pandas UDF acceptable?
seniorWholeStageCodegen is off for a stage. How do you find why?
Tungsten execution engine · tap to open the answer
Short: An operator that can't be compiled — often a UDF, an unsupported expression, or a fallback.
Detailed: explain() and SQL UI: look for the break in the * codegen star. Fix the expression. Don't 'tune Tungsten' as a config cult.
Common mistake: Random spark.tungsten flags from a blog.
Follow-up: How does Photon relate to Tungsten on Databricks?