RDDs: lineage and partitions
Legacy RDD job 8× slower than the DataFrame rewrite
Symptoms
- Same cluster
- mapPartitions with Python
RDDs: lineage and partitions interview questions
Interactive Spark interview questions on RDDs: lineage and partitions. Same topic as /learn/spark/rdd. The original Spark API. An RDD is a partitioned collection with lineage so lost partitions can be recomputed.
Question 1 of 3
beginner
RDDs: lineage and partitionsWhat is an RDD, and why do we still mention it?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is an RDD, and why do we still mention it?
RDDs: lineage and partitions · tap to open the answer
Short: The original distributed collection. DataFrames compile down toward RDD/Tungsten execution.
Detailed: You rarely write RDD code now. Interviewers want: partitions, lineage, and that DataFrames are the API you should use.
Common mistake: Starting a new ETL in RDD map/reduce.
Follow-up: What is lineage on an RDD?
intermediateWhy is Dataset/DataFrame usually faster than hand-rolled RDDs?
RDDs: lineage and partitions · tap to open the answer
Short: Catalyst optimizes the plan; Tungsten uses off-heap/binary rows. RDDs are Java objects.
Detailed: RDD map on Row objects boxes everything. DataFrame expressions can whole-stage codegen. You also get predicate pushdown on files.
Common mistake: rdd.filter because 'it's more control'.
Follow-up: When is an RDD the right tool?
seniorYou need a custom partitioner for a skewed key. Do you drop to RDD?
RDDs: lineage and partitions · tap to open the answer
Short: Sometimes — RDD partitionBy with a custom Partitioner, or stay in DataFrame with salting.
Detailed: Prefer salting + AQE in DataFrame land. Custom partitioners are powerful and easy to get wrong (breaking join co-partitioning). Measure first.
Common mistake: Custom partitioner as the first skew fix.
Follow-up: What join property do you lose if only one side is custom-partitioned?