Interview/Spark

Spark interview questions: UDFs: the box Catalyst cannot see into

Interactive Spark interview questions on UDFs: the box Catalyst cannot see into. Practice with the matching lesson. A Python UDF is a black box Catalyst cannot optimize. Built-ins, higher-order functions, then pandas_udf.

Lesson · Simulation

You wrote a Python UDF to uppercase a column. Why does a reviewer push back?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

UDFs: the box Catalyst cannot see into

Nightly job went from 12 minutes to 5 hours after a one-line phone-number cleanup UDF

Symptoms

  • The only change in the PR was a Python UDF on one column
  • Input Size / Records on the scan jumped by two orders of magnitude
  • Executor CPU is mostly in Python worker processes

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

You wrote a Python UDF to uppercase a column. Why does a reviewer push back?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: Every row leaves the JVM for a Python process and comes back; upper() never leaves the JVM at all.

Why: A Python UDF serializes rows to a Python worker, evaluates them one at a time, and serializes results back — it shows up as BatchEvalPython in the plan. Built-ins like upper, regexp_replace, and to_timestamp compile into whole-stage codegen and operate on Tungsten binary rows directly.

Common mistake: Assuming the UDF is fine 'because it's just string formatting'.

Follow-up they will ask: Name a built-in you would use instead of a UDF you have written before.

Lesson · Simulation

intermediate

You moved a filter into a UDF and the scan went from 4 GB to 900 GB. Explain.

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: Catalyst cannot see inside a UDF, so the predicate is not pushed into the scan and no partitions get pruned.

Why: A UDF is an opaque expression: it can never become a PushedFilter or a PartitionFilter, so Spark reads every file and filters afterwards above BatchEvalPython. explain('formatted') shows PushedFilters empty and the FileScan's Input Size / Records covering the whole table.

Common mistake: Blaming 'Python is slow' when the real cost is the extra 896 GB of I/O.

Follow-up they will ask: How do you keep the custom logic but restore pruning?

Lesson · Simulation

senior

Walk me up the fix ladder for a row-at-a-time Python UDF, cheapest first.

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: Built-in expression, then higher-order functions, then pandas_udf, and only then a plain Python UDF.

Why: Built-ins keep codegen and pushdown. Higher-order functions (transform, filter, exists, aggregate) handle array and map logic without leaving SQL. pandas_udf ships columnar Arrow batches, so you pay serialization per batch instead of per row and get vectorized pandas or NumPy code. A row-wise Python UDF is the last resort.

Senior depth: It is still opaque to Catalyst, so nothing pushes through it, and on Databricks it still knocks the query off Photon. Faster, not free.

Common mistake: Jumping straight to pandas_udf when composing two built-ins would have done it.

Follow-up they will ask: What does a pandas UDF still not fix?

Lesson · Simulation

architect

A Scala UDF runs inside the JVM, so is it safe to use anywhere?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It skips the Python round trip, but it is still a black box to the optimizer.

Why: A Scala or Java UDF executes in the executor JVM with no cross-process serialization, so throughput is far better than Python. Catalyst still cannot rewrite or push it, it can break whole-stage codegen for the operators around it, and null handling is entirely your responsibility. A native Expression or a composition of built-ins is still preferable when the logic can be expressed.

Senior depth: Rerun the query with that column replaced by a literal. If runtime collapses, it is the UDF; if not, you were about to optimize the wrong operator.

Common mistake: Treating 'we made it a Scala UDF' as equivalent to a native expression in a design review.

Follow-up they will ask: How do you prove the UDF is or is not your bottleneck?

Lesson · Simulation

beginner

You registered your UDF with spark.udf.register and call it from SQL instead of the DataFrame API. Does that change how it runs?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: No — registration only makes it callable by name in SQL; the execution path and the optimizer blindness are identical.

Remember these

  • Registration affects name resolution, not the execution path
  • The plan still shows BatchEvalPython either way
  • Catalyst remains unable to push filters through the function
  • SQL registration widens the blast radius by making it easy to reuse

Why: spark.udf.register adds the function to the session's function registry so a SQL string can resolve the name. The resulting expression is the same opaque Python UDF node, so the plan still shows BatchEvalPython, rows are still pickled out to a Python worker and back, and Catalyst still cannot push a predicate through it or fold it into whole-stage codegen. The only real difference is who can call it: analysts writing SQL can now use it, which in practice makes the performance problem spread further rather than shrink. Compare it with a built-in SQL function, which resolves to a Catalyst Expression with a codegen implementation and lives entirely inside the JVM.

spark.udf.register("clean_phone", clean_phone, StringType())
spark.sql("SELECT clean_phone(phone) FROM customers").explain("formatted")

Common mistake: Believing a SQL-registered UDF gets optimized like a built-in because it is written in SQL syntax.

Follow-up they will ask: Is a Spark SQL function defined with CREATE FUNCTION in SQL the same thing?

Lesson · Simulation

intermediate

Your plan shows BatchEvalPython and a colleague's shows ArrowEvalPython. What changed in the code, and what does it change at runtime?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: They moved to a pandas_udf, so rows cross to Python as Arrow columnar batches instead of one pickled row at a time.

Remember these

  • BatchEvalPython crosses per row; ArrowEvalPython crosses per batch
  • pandas_udf receives and returns a pandas Series, so the body vectorizes
  • maxRecordsPerBatch controls how many rows cross at a time
  • Arrow makes it faster, not transparent — pushdown is still blocked

Why: BatchEvalPython is the plain Python UDF path: each row is serialized with pickle, written over a socket to a Python worker, evaluated by CPython, and serialized back. ArrowEvalPython is what a pandas_udf plans to: Spark converts a batch of rows into an Arrow record batch, hands the whole batch over once, and your function receives a pandas Series and returns a pandas Series. You pay the boundary crossing per batch instead of per row, and the function body runs vectorized in NumPy or pandas rather than in a Python loop. Getting there means declaring the function with pandas type hints and a pandas_udf decorator; the relevant knobs are spark.sql.execution.arrow.pyspark.enabled and spark.sql.execution.arrow.maxRecordsPerBatch, which sets how many rows go over at once. What does not change is that it is still an opaque expression — nothing pushes through an ArrowEvalPython node either.

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def to_usd(amount: pd.Series, rate: pd.Series) -> pd.Series:
    return amount * rate

Common mistake: Writing a pandas_udf whose body still loops row by row over the Series, which keeps the Python cost and only saves serialization.

Follow-up they will ask: What would make you raise or lower maxRecordsPerBatch?

Lesson · Simulation

advanced

A UDF calls an external REST API once per row. Beyond speed, what is wrong with that, and how do you restructure it?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It is nondeterministic and retried unpredictably, and it fans out uncontrolled concurrency at the remote service — batch the calls and bound the parallelism.

Remember these

  • Spark may evaluate an expression more than once; side effects are not safe
  • Task count becomes your concurrency against the remote service
  • Batch with mapInPandas and set the partition count deliberately
  • Materializing the lookup into a table turns the call into a broadcast join

Why: Spark assumes expressions are pure and may evaluate them more than once: a task retry, speculative execution, or an optimizer rewrite can all cause a second call, so duplicate side effects and inconsistent results are structural rather than unlucky. Concurrency is the second problem — every task runs the UDF simultaneously, so a 400-task stage means up to 400 concurrent callers hitting a service that was sized for far less, and your own job causes the rate limiting it then fails on. The restructure is to stop working per row: use mapInPandas or a pandas_udf so each call handles a batch, control the fan-out by setting the partition count deliberately rather than inheriting it, and add retry and backoff inside the function because Spark's task-level retry is far too coarse. The better answer when it is available is to not call the service in the data path at all — materialize the lookup into a table on a schedule and turn the enrichment into a broadcast join.

def enrich(batches):
    for pdf in batches:
        pdf["geo"] = lookup_many(pdf["ip"].tolist())   # one call per batch
        yield pdf

df.repartition(32).mapInPandas(enrich, schema=out_schema)

Senior depth: Handle the failure inside the function rather than letting it fail the task, since a task failure reprocesses the entire partition and re-issues every call in it. Catch per-item errors, emit a null or an error column for the failed items, and let a downstream step reprocess that subset. The design principle is to make the retry granularity match the call granularity.

Common mistake: Fixing the slowness by increasing executor count, which multiplies the load on the service that was already the bottleneck.

Follow-up they will ask: How would you make the batched version resilient to a partial failure inside one batch?

Lesson · Simulation

senior

A Python UDF is declared to return IntegerType and about 2% of its output comes back null, with no errors anywhere. Explain.

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: When a returned value does not fit the declared return type, PySpark yields null instead of raising.

Remember these

  • A return value that does not match the declared type becomes null, not an error
  • Scala UDFs with primitive parameters turn input nulls into 0 or false
  • Diagnose by selecting rows where output is null and input is not
  • Use nullable types and explicit edge-case handling inside the function

Why: The return type you pass to udf() is a contract Spark applies to whatever Python hands back. If the function returns a float, a string, or a value outside the integer range for those 2% of rows, the conversion fails and the result is null rather than an exception — silent data loss that looks like upstream nulls. The same category of trap exists on the input side for JVM UDFs: a Scala UDF with primitive parameters receives 0 or false where the column was null, because a primitive cannot hold null, so your logic sees a value that was never in the data. The diagnosis is to filter to the rows where the output is null and the input is not, then run the function on those inputs in plain Python to see what it actually produced. The fixes are to widen or correct the declared return type, to handle the edge cases explicitly inside the function, and to use boxed or Option types in Scala UDFs so nulls stay null.

bad = df.where(F.col("score").isNull() & F.col("raw").isNotNull())
bad.select("raw").show(20, truncate=False)

Senior depth: Assert on it. Add a post-transformation check that the null rate of the derived column is within an expected band and fail the job when it is not, and unit-test the Python function directly against the known edge cases rather than only testing the DataFrame end to end. Silent null production is the characteristic UDF failure, so the test has to be about null rate, not about the happy path.

Common mistake: Blaming the source data for nulls that the UDF's own type contract created.

Follow-up they will ask: How would you catch this class of bug before it reaches production?

Lesson · Simulation

senior

When does asNondeterministic() actually matter, and what does it not fix?

UDFs: the box Catalyst cannot see into · tap to open the answer

Say this first: It stops Catalyst from reordering, pushing, or collapsing the UDF — but it does not make repeated evaluation produce the same answer.

Remember these

  • Catalyst assumes determinism and may reorder or duplicate the UDF
  • asNondeterministic pins it in place and prevents those rewrites
  • It does not make task retries reproduce the same values
  • For reproducibility, derive from a column seed or materialize the values

Why: By default Catalyst treats a UDF as deterministic and is therefore free to push it below or above a filter, duplicate it into both branches of a plan, or collapse adjacent projections that reference it. For a function involving randomness, the current time, or an external call, those rewrites change results — for example a random-sampling UDF pushed beneath a filter is evaluated on a different row set than you wrote. Marking it with asNondeterministic() tells the optimizer to leave it where it is and evaluate it once per row, which restores the semantics you intended. What it does not do is guarantee stability across retries: if a task fails and reruns, the function runs again and produces different values, and any already-committed downstream output can disagree. For reproducibility you need the nondeterminism removed from the data path — seed it from a column, or materialize the generated values into a table and join against them.

import random
from pyspark.sql.functions import udf
sample_flag = udf(lambda: random.random() < 0.01, "boolean").asNondeterministic()

Senior depth: rand() is marked nondeterministic in the optimizer but takes a seed, and the seed is combined with the partition index — so the same plan over the same partitioning gives the same values. That is reproducible only as long as the partitioning is stable, which it is not across a repartition or a change in shuffle partition count. If reproducibility genuinely matters, materialize the generated column once and read it back.

Common mistake: Reaching for asNondeterministic to fix inconsistent results across runs, which is a retry problem it cannot touch.

Follow-up they will ask: Where does Spark's own rand() sit on this spectrum?

Lesson · Simulation