Pandas prototype copied into Spark
Symptoms
- for row in df.collect()
- UDF per row
Interactive Spark interview questions on DataFrames and schemas. Practice with the matching lesson. A DataFrame is a distributed table with a schema. Catalyst can optimize it; an RDD of tuples cannot.
Question 1 of 8
What is a DataFrame in Spark?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 8 of 8 questions.
What is a DataFrame in Spark?
DataFrames and schemas · tap to open the answer
Say this first: A distributed table with a schema and a lazy query plan.
Why: Rows are partitioned across executors. Operations return a new plan, not a local pandas object. spark.createDataFrame on a huge Python list still starts on the driver.
Common mistake: Treating it like pandas (df[i] loops, collect to 'feel the data').
Follow-up they will ask: How do you print the schema without executing a job?
When should you drop to RDDs from a DataFrame?
DataFrames and schemas · tap to open the answer
Say this first: Almost never for ETL. DataFrames get Catalyst and Tungsten. RDDs skip that.
Why: Use RDDs only for custom partitioning or APIs that still need them. groupBy/join/filter belong on DataFrames/Datasets.
Common mistake: rdd.map for a column expression Catalyst could optimize.
Follow-up they will ask: What do you lose when you call rdd.map on a DataFrame?
createDataFrame(huge_python_list) OOMs before any executor works. Why?
DataFrames and schemas · tap to open the answer
Say this first: The list lives on the driver; Spark then has to ship it.
Why: Parallelize/createDataFrame from driver memory. Read from storage or a Spark table instead. Same class of bug as collect() in reverse.
Common mistake: Raising executor memory for a driver-side Python list.
Follow-up they will ask: How should a 50 GB CSV enter Spark?
df2 = df.withColumn('x', lit(1)) gives you a new DataFrame. Did Spark copy the data?
DataFrames and schemas · tap to open the answer
Say this first: No — it appended a node to a query plan. No rows exist yet for either variable.
Remember these
Why: A DataFrame is a handle to an immutable logical plan, so every transformation returns a new handle wrapping the old plan plus one operator. Nothing is allocated per row, which is why chaining fifty transformations on a petabyte-scale table costs microseconds. The two variables are not two datasets: df2's plan contains df's plan as a subtree, and if you run an action on each one, Spark executes the shared part twice unless you cache it. Immutability is what lets Catalyst rewrite freely — it can reorder a filter below a projection because no one holds a mutable reference to an intermediate state. Call df2.explain() and you see a Project node over whatever df was; call df2.count() and only then does a job appear.
df2 = df.withColumn('x', lit(1))
df2.explain() # Project over the original plan, zero jobsCommon mistake: Reasoning about DataFrames like pandas objects and worrying about the memory cost of 'another copy'.
Follow-up they will ask: If both df and df2 are used in actions, what have you paid twice?
Two teams union their DataFrames and the output has amounts in the region column. Both schemas 'match'. What happened?
DataFrames and schemas · tap to open the answer
Say this first: union() matches columns by position, not by name, so a reordered upstream schema silently swaps values.
Remember these
Why: DataFrame.union is the SQL UNION ALL contract: it lines up the Nth column of the left side with the Nth column of the right side and only checks that the count matches and the types are compatible. If both sides happen to have a string and a decimal in different orders, or if an upstream table added a column in the middle, the plan is still valid and no error is raised. Use unionByName when the inputs are produced independently; it resolves by column name and fails loudly on a mismatch. With allowMissingColumns=True it fills absent columns with null instead of failing, which is the right tool for schema evolution but hides genuinely missing data if you reach for it reflexively. The plan node is Union either way, so you cannot tell from the physical plan which semantics you got — only the API call tells you. Compare df.columns on both sides in a test and the bug never ships.
a.unionByName(b) # strict, name-aligned
a.unionByName(b, allowMissingColumns=True) # evolution-tolerantCommon mistake: Assuming union validates names because the first version of both tables happened to be in the same order.
Follow-up they will ask: How would a unit test catch this before production?
You filter a Parquet table on amount > 100. How do you prove the filter reached the scan instead of running above it?
DataFrames and schemas · tap to open the answer
Say this first: Read explain('formatted') and check PushedFilters on the FileScan node, then confirm Input Size / Records dropped in the UI.
Remember these
Why: The formatted plan prints three separate lists on a file scan: PartitionFilters for predicates that eliminate whole directories, PushedFilters for predicates handed to the Parquet reader, and DataFilters for predicates Spark still evaluates itself. A predicate in PushedFilters lets the reader skip row groups whose min/max statistics cannot match, so the win shows up as reduced Input Size / Records on the Stages page, not as a changed plan shape. If the predicate appears only in a separate Filter node above the scan, every row was read and discarded. Common reasons pushdown fails: the predicate wraps the column in a function or a cast, it compares against a non-deterministic expression, it is an OR spanning pushable and non-pushable terms, or the whole thing is hidden inside a UDF that the optimizer cannot see into. Verify spark.sql.parquet.filterPushdown is still on in the Environment tab before blaming the query. The two-step habit — plan first, then Input Size — is what turns a guess into evidence.
df.filter('amount > 100').explain('formatted')
# FileScan parquet ... PushedFilters: [IsNotNull(amount), GreaterThan(amount,100)]Common mistake: Declaring pushdown works because the query got faster, without checking Input Size / Records.
Follow-up they will ask: Why does a filter on cast(id as string) usually not push down?
After a currency conversion and a sum, a decimal revenue column comes back as nulls. What is the mechanism?
DataFrames and schemas · tap to open the answer
Say this first: Decimal arithmetic promotes precision and scale, and when the result needs more precision than decimal's 38-digit maximum, Spark returns null instead of raising by default.
Remember these
Why: Every decimal operation derives a result type from its inputs: multiplication adds the precisions and the scales, and division inflates scale further. Chain a few operations on decimal(38,18) values and the derived type exceeds the 38-digit limit, so Spark has to reduce it. With spark.sql.decimalOperations.allowPrecisionLoss enabled, it keeps precision at 38 and trims scale, which quietly rounds; when a value genuinely cannot be represented, the result becomes null rather than an error unless spark.sql.ansi.enabled is on, in which case you get an overflow exception instead. That is why a revenue column turns to nulls after a step that looks arithmetically harmless, and why the nulls appear only for the largest values. The diagnosis is to print the schema after each step and watch the decimal type inflate. The fix is to control the type deliberately: cast to a scale you actually need before multiplying, or use double where you are summing approximate money-free quantities.
df.select((col('amt') * col('fx')).alias('rev')).printSchema()
df.select(col('amt').cast('decimal(20,4)') * col('fx').cast('decimal(12,6)'))Senior depth: Silent null on overflow is the worst default for financial data because it is indistinguishable from a missing value. On money pipelines I pin explicit decimal types at each boundary and run with ANSI mode so overflow is an incident instead of a slowly wrong dashboard.
Common mistake: Blaming the upstream source for nulls that Spark's own decimal promotion produced.
Follow-up they will ask: What does turning on spark.sql.ansi.enabled change about this failure?
Your Parquet files declare a column as required, Spark reads it as nullable, and a downstream assertion breaks. Why does Spark do that?
DataFrames and schemas · tap to open the answer
Say this first: Spark treats nullability as an optimizer hint rather than an enforced constraint, and readers widen to nullable to stay safe across files.
Remember these
Why: The nullable flag on a StructField tells Catalyst it may skip null handling in generated code — it is not a validated constraint on the data. Because a scan can span many files with slightly different writers, the Parquet source generally reports columns as nullable so that one file with nulls cannot produce wrong results or a crash in codegen generated under the opposite assumption. Operations also legitimately widen nullability: the right side of a left outer join becomes nullable, and so does any column under a window or an aggregate that can see empty groups. If you force nullable=False on a schema that actually contains nulls, you are lying to the optimizer, and the symptom is not a clean error but wrong results or a NullPointerException deep in generated code. The place to enforce 'this is never null' is the storage layer — a Delta NOT NULL constraint or a CHECK constraint is validated on write, every write, by every writer. Assert on data with a count of nulls, not on the schema flag.
spark.sql("ALTER TABLE silver.orders ALTER COLUMN order_id SET NOT NULL")
df.filter(col('order_id').isNull()).count() # assert on data, not on the flagSenior depth: Schema flags describe what the optimizer is allowed to assume; constraints describe what the data is allowed to be. Teams that conflate the two end up with schemas that assert guarantees nothing validates, and the failure surfaces as a codegen NullPointerException rather than a data quality alert.
Common mistake: Forcing nullable=False in a hand-built StructType to make a downstream check pass.
Follow-up they will ask: Which join type changes nullability on which side?