Interview/Spark

Spark interview questions: Tungsten execution engine

Interactive Spark interview questions on Tungsten execution engine. Practice with the matching lesson. Tungsten stores rows off-heap and generates JVM bytecode for whole stages so Spark SQL is not a naive iterator of JVM objects.

Lesson · Simulation

What is Tungsten?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Tungsten execution engine

Same SQL 5× slower after a 'tiny' UDF

Symptoms

  • One email-normalize Python UDF
  • Rest of the query native

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What is Tungsten?

Tungsten execution engine · tap to open the answer

Say this first: Spark's execution engine: binary rows, off-heap memory, whole-stage codegen.

Why: It avoids Java object overhead on the hot path. You get it with DataFrame/SQL. Python UDFs jump back to objects and kill the benefit.

Common mistake: Tungsten as a Databricks SKU.

Follow-up they will ask: What operator in explain() shows whole-stage codegen?

Lesson · Simulation

intermediate

Why does a Python UDF disable the fast path?

Tungsten execution engine · tap to open the answer

Say this first: Rows must be deserialized into Python, one call at a time (or batches for pandas UDFs).

Why: Plan shows BatchEvalPython / PythonUDF. Photon also bails out. Rewrite with Spark functions or Scala.

Common mistake: Pandas UDF as 'basically native'.

Follow-up they will ask: When is a pandas UDF acceptable?

Lesson · Simulation

senior

WholeStageCodegen is off for a stage. How do you find why?

Tungsten execution engine · tap to open the answer

Say this first: An operator that can't be compiled — often a UDF, an unsupported expression, or a fallback.

Why: explain() and SQL UI: look for the break in the * codegen star. Fix the expression. Don't 'tune Tungsten' as a config cult.

Common mistake: Random spark.tungsten flags from a blog.

Follow-up they will ask: How does Photon relate to Tungsten on Databricks?

Lesson · Simulation

intermediate

In explain("formatted") some operators carry a prefix like *(2). What is the star, and what does the number mean?

Tungsten execution engine · tap to open the answer

Say this first: The star marks operators compiled into whole-stage codegen; the number is the codegen stage they were fused into.

Remember these

  • *(n) means this operator was fused into generated codegen stage n
  • An operator with no star fell off the compiled fast path
  • Exchange is never part of a codegen stage

Why: Whole-stage codegen takes a run of operators and generates a single Java method that loops over rows, instead of calling next() between operator objects. Everything sharing the same *(n) was fused into generated stage n, so the CPU runs one tight loop over Tungsten binary rows with no virtual dispatch and no intermediate objects. An operator printed without a star was not compiled — a Python UDF, an unsupported expression, or a fallback — and rows have to be materialized across that boundary. The number increments per codegen stage, so a plan with *(1) through *(5) has five fused regions, typically separated by Exchange nodes, which are never part of a codegen stage. Reading the stars is the fastest way to spot where the fast path broke.

df.groupBy("region").sum("amount").explain("formatted")
# *(1) HashAggregate and *(2) HashAggregate around the Exchange

Common mistake: Reading the number as a stage of the job rather than a codegen stage of the plan.

Follow-up they will ask: Which operator in your last query lost its star?

Lesson · Simulation

advanced

What does whole-stage codegen actually generate, and why does fusing operators help the CPU rather than only saving function calls?

Tungsten execution engine · tap to open the answer

Say this first: It generates one Java method with the operator chain inlined as a loop, so values stay in local variables instead of travelling through boxed rows.

Remember these

  • One generated method replaces a chain of iterator next() calls
  • Values stay in local variables instead of boxed row objects
  • Sequential reads over binary rows make CPU caches effective
  • explain("codegen") prints the generated Java source

Why: Without codegen each operator is an iterator that pulls a row from its child through a virtual call, which defeats branch prediction and forces the row to be materialized at every step. Codegen emits one method where filter conditions, projections, and aggregate updates are inlined into a loop over the input, so intermediate values live in locals the JIT can keep in registers. Combined with Tungsten's binary row format — fixed-width fields at known offsets in a contiguous buffer — the loop reads memory sequentially, which is what makes hardware prefetching and cache lines pay off. You can read the generated source with df.explain("codegen"). The limit is method size: HotSpot will not JIT-compile a method past roughly 64 KB of bytecode, so very long operator chains have to fall back.

df.filter("amount > 0").select("id").explain("codegen")

Senior depth: spark.sql.codegen.hugeMethodLimit caps the bytecode size Spark will emit for a codegen stage, because past the JVM's method size limit HotSpot refuses to JIT and the method would run interpreted — slower than the non-codegen path. When a stage exceeds it, Spark falls back to the iterator model for that stage. spark.sql.codegen.maxFields bounds how many schema fields a codegen stage will handle, which is why very wide tables lose their stars with no UDF involved.

Common mistake: Describing codegen as 'avoiding function call overhead' and stopping there, missing the memory-layout half.

Follow-up they will ask: What makes a generated method too big to compile?

Lesson · Simulation

advanced

A stage lost whole-stage codegen even though the plan contains no UDF. What causes would you check?

Tungsten execution engine · tap to open the answer

Say this first: Too many schema fields, a generated method past the size limit, a compile failure that fell back, or an operator with no codegen support.

Remember these

  • Wide schemas trip spark.sql.codegen.maxFields
  • Long chains trip spark.sql.codegen.hugeMethodLimit
  • A compile failure falls back silently and only shows in executor logs

Why: spark.sql.codegen.maxFields bounds the number of schema fields a codegen stage will handle, so a very wide table can disable it by width alone. spark.sql.codegen.hugeMethodLimit bounds generated bytecode size, and a long operator chain or a huge CASE expression can exceed it, at which point that stage runs through the iterator path. The code generator can also fail to compile the generated source, and with fallback enabled Spark logs the failure and runs interpreted instead of failing the query — so check executor logs for the compile error rather than assuming your plan is at fault. Finally, some operators have no codegen implementation and are always unstarred. The fix is almost always to narrow the projection or shorten the chain, not to flip Tungsten configs.

spark.conf.get("spark.sql.codegen.maxFields")
spark.conf.get("spark.sql.codegen.hugeMethodLimit")

Senior depth: Read executor stderr for the codegen compile exception and the generated source dump — Spark logs the failing class. Reproduce with a narrowed projection to find which expression blows the method up, then split the query at that point so each half compiles. Setting spark.sql.codegen.wholeStage to false globally is a legitimate experiment and a terrible fix.

Common mistake: Assuming any missing star means a UDF, and hunting for one that is not there.

Follow-up they will ask: How would you confirm a compile failure rather than guess at it?

Lesson · Simulation

senior

Make the memory and garbage-collection argument for UnsafeRow, not the CPU one.

Tungsten execution engine · tap to open the answer

Say this first: A binary row is one contiguous buffer with no per-field object header, so it uses far less space and the collector has almost nothing to trace.

Remember these

  • One buffer per row, with no object header per field
  • The collector traces one object instead of one per field
  • Off-heap execution memory removes that memory from GC entirely
  • Off-heap still counts against the container's total limit

Why: A row modelled as Java objects costs an object header per field plus pointers, and a String carries a character array of its own, so the in-memory footprint can be several times the logical data size. UnsafeRow lays fields out at fixed offsets in a single byte buffer — null bits, then fixed-width values, then a variable-length region — so one row is one allocation. That also means the garbage collector traces one object instead of thousands, which is what keeps GC time flat when a task holds millions of rows in a hash aggregation. Turning on spark.memory.offHeap.enabled with spark.memory.offHeap.size moves those buffers out of the heap entirely, so execution memory is managed by Spark rather than by the collector. The caveat is that off-heap memory still counts against the container's total limit, so it trades GC pressure for careful overhead budgeting.

spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "4g")

Senior depth: They are converted to Arrow record batches, shipped to a Python worker, and rebuilt as pandas objects, so you pay the serialization and hold both representations at once. That memory lives in the Python process, outside the JVM heap, which is why those jobs are killed by the cluster manager while Spark's heap metrics look healthy. Budget spark.executor.memoryOverhead for it and keep the Arrow batch size modest.

Common mistake: Treating Tungsten as purely a CPU story, then being surprised a Python UDF raises memory as well as latency.

Follow-up they will ask: What happens to those rows when a pandas UDF runs?

Lesson · Simulation

architect

Where does the vectorized Parquet reader sit relative to Tungsten, and when does it stop being used?

Tungsten execution engine · tap to open the answer

Say this first: It fills Tungsten's columnar batches straight from Parquet pages, and it is bypassed for schemas it cannot handle.

Remember these

  • The vectorized reader produces ColumnarBatch that codegen'd scans read directly
  • spark.sql.parquet.enableVectorizedReader gates it
  • Unsupported schemas fall back to a row-based read
  • Flat schemas keep the whole read path in binary form

Why: The vectorized reader decodes Parquet column chunks into ColumnarBatch objects — a few thousand rows at a time in columnar buffers — instead of constructing one row at a time, and codegen'd scans consume those batches directly. That keeps the whole read path in the binary world: no Java object per value, dictionary encoding preserved where possible, and sequential memory access. It is gated by spark.sql.parquet.enableVectorizedReader and bounded by spark.sql.parquet.columnarReaderBatchSize, and it is skipped for schemas it does not support, historically deeply nested structures, in which case the plan falls back to a row-based read. Architecturally that is an argument for flat, typed schemas at the silver layer: nested JSON costs you the vectorized path before any of your query logic runs.

spark.conf.get("spark.sql.parquet.enableVectorizedReader")
spark.conf.get("spark.sql.parquet.columnarReaderBatchSize")

Senior depth: Check whether the scan node reports columnar output, and compare scan time per byte against a flattened copy of the same data. The practical test is to project only the flat columns: if the scan speeds up far more than the byte reduction explains, the nested column was costing you the vectorized path. Then fix it at ingestion by flattening in bronze rather than per query.

Common mistake: Blaming Parquet for a slow scan when the schema pushed the read off the vectorized path.

Follow-up they will ask: How would you prove a table lost the vectorized reader?

Lesson · Simulation