DataFrames and schemas
Pandas prototype copied into Spark
Symptoms
- for row in df.collect()
- UDF per row
DataFrames and schemas interview questions
Interactive Spark interview questions on DataFrames and schemas. Same topic as /learn/spark/dataframe. A DataFrame is a distributed table with a schema. Catalyst can optimize it; an RDD of tuples cannot.
Question 1 of 3
beginner
DataFrames and schemasWhat is a DataFrame in Spark?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is a DataFrame in Spark?
DataFrames and schemas · tap to open the answer
Short: A distributed table with a schema and a lazy query plan.
Detailed: Rows are partitioned across executors. Operations return a new plan, not a local pandas object. spark.createDataFrame on a huge Python list still starts on the driver.
Common mistake: Treating it like pandas (df[i] loops, collect to 'feel the data').
Follow-up: How do you print the schema without executing a job?
intermediateWhen should you drop to RDDs from a DataFrame?
DataFrames and schemas · tap to open the answer
Short: Almost never for ETL. DataFrames get Catalyst and Tungsten. RDDs skip that.
Detailed: Use RDDs only for custom partitioning or APIs that still need them. groupBy/join/filter belong on DataFrames/Datasets.
Common mistake: rdd.map for a column expression Catalyst could optimize.
Follow-up: What do you lose when you call rdd.map on a DataFrame?
seniorcreateDataFrame(huge_python_list) OOMs before any executor works. Why?
DataFrames and schemas · tap to open the answer
Short: The list lives on the driver; Spark then has to ship it.
Detailed: Parallelize/createDataFrame from driver memory. Read from storage or a Spark table instead. Same class of bug as collect() in reverse.
Common mistake: Raising executor memory for a driver-side Python list.
Follow-up: How should a 50 GB CSV enter Spark?