Say this first: It stops Catalyst from reordering, pushing, or collapsing the UDF — but it does not make repeated evaluation produce the same answer.
Remember these
- Catalyst assumes determinism and may reorder or duplicate the UDF
- asNondeterministic pins it in place and prevents those rewrites
- It does not make task retries reproduce the same values
- For reproducibility, derive from a column seed or materialize the values
Why: By default Catalyst treats a UDF as deterministic and is therefore free to push it below or above a filter, duplicate it into both branches of a plan, or collapse adjacent projections that reference it. For a function involving randomness, the current time, or an external call, those rewrites change results — for example a random-sampling UDF pushed beneath a filter is evaluated on a different row set than you wrote. Marking it with asNondeterministic() tells the optimizer to leave it where it is and evaluate it once per row, which restores the semantics you intended. What it does not do is guarantee stability across retries: if a task fails and reruns, the function runs again and produces different values, and any already-committed downstream output can disagree. For reproducibility you need the nondeterminism removed from the data path — seed it from a column, or materialize the generated values into a table and join against them.
import random
from pyspark.sql.functions import udf
sample_flag = udf(lambda: random.random() < 0.01, "boolean").asNondeterministic()
Senior depth: rand() is marked nondeterministic in the optimizer but takes a seed, and the seed is combined with the partition index — so the same plan over the same partitioning gives the same values. That is reproducible only as long as the partitioning is stable, which it is not across a repartition or a change in shuffle partition count. If reproducibility genuinely matters, materialize the generated column once and read it back.
Common mistake: Reaching for asNondeterministic to fix inconsistent results across runs, which is a retry problem it cannot touch.
Follow-up they will ask: Where does Spark's own rand() sit on this spectrum?