Interview/Spark

Spark interview questions: RDDs: lineage and partitions

Interactive Spark interview questions on RDDs: lineage and partitions. Practice with the matching lesson. The original Spark API. An RDD is a partitioned collection with lineage so lost partitions can be recomputed.

Lesson · Simulation

What is an RDD, and why do we still mention it?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

RDDs: lineage and partitions

Legacy RDD job 8× slower than the DataFrame rewrite

Symptoms

  • Same cluster
  • mapPartitions with Python

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What is an RDD, and why do we still mention it?

RDDs: lineage and partitions · tap to open the answer

Say this first: The original distributed collection. DataFrames compile down toward RDD/Tungsten execution.

Why: You rarely write RDD code now. Interviewers want: partitions, lineage, and that DataFrames are the API you should use.

Common mistake: Starting a new ETL in RDD map/reduce.

Follow-up they will ask: What is lineage on an RDD?

Lesson · Simulation

intermediate

Why is Dataset/DataFrame usually faster than hand-rolled RDDs?

RDDs: lineage and partitions · tap to open the answer

Say this first: Catalyst optimizes the plan; Tungsten uses off-heap/binary rows. RDDs are Java objects.

Why: RDD map on Row objects boxes everything. DataFrame expressions can whole-stage codegen. You also get predicate pushdown on files.

Common mistake: rdd.filter because 'it's more control'.

Follow-up they will ask: When is an RDD the right tool?

Lesson · Simulation

senior

You need a custom partitioner for a skewed key. Do you drop to RDD?

RDDs: lineage and partitions · tap to open the answer

Say this first: Sometimes — RDD partitionBy with a custom Partitioner, or stay in DataFrame with salting.

Why: Prefer salting + AQE in DataFrame land. Custom partitioners are powerful and easy to get wrong (breaking join co-partitioning). Measure first.

Common mistake: Custom partitioner as the first skew fix.

Follow-up they will ask: What join property do you lose if only one side is custom-partitioned?

Lesson · Simulation

beginner

groupByKey or reduceByKey for a per-key sum on an RDD, and why?

RDDs: lineage and partitions · tap to open the answer

Say this first: reduceByKey — it combines values on the map side, so far fewer bytes cross the shuffle.

Remember these

  • reduceByKey combines on the map side before the shuffle
  • groupByKey ships every raw value and can overwhelm one reducer
  • Compare Shuffle Write bytes to see the difference
  • DataFrame groupBy inserts partial aggregation automatically

Why: groupByKey ships every value for a key to the reducer that owns it and hands you an iterable, so the shuffle carries the raw dataset and one hot key means one reducer holding all of its values in memory. reduceByKey applies your associative function locally within each map partition first, so a partition with ten thousand rows for a key sends one partial sum instead of ten thousand values. Shuffle Write bytes in the UI show the difference directly, usually by an order of magnitude on skewed data. The combine step is also what keeps a single large key from blowing up a reducer, because the per-key payload is already reduced before it travels. aggregateByKey and combineByKey are the general forms for when the accumulator type differs from the value type. The DataFrame equivalent is a groupBy with an aggregate, where Catalyst inserts the partial aggregation for you — one more reason not to hand-roll this in RDDs.

rdd.reduceByKey(lambda a, b: a + b)          # partial sums shuffle
# rdd.groupByKey().mapValues(sum)             # whole dataset shuffles

Common mistake: Using groupByKey because the code reads more naturally, then blaming skew for the reducer OOM.

Follow-up they will ask: When is groupByKey genuinely the right choice?

Lesson · Simulation

intermediate

What does a single df.rdd.map(...) cost you in the middle of a DataFrame pipeline?

RDDs: lineage and partitions · tap to open the answer

Say this first: It converts optimised binary rows into JVM or Python objects, ends codegen, and leaves everything downstream outside Catalyst's reach.

Remember these

  • DeserializeToObject converts binary rows into real objects
  • Codegen fusion ends at the RDD boundary
  • Pushdown and pruning cannot cross back over the conversion
  • PySpark adds pickling and a python worker hop per row

Why: Inside the DataFrame API, rows are Tungsten InternalRow in off-heap-friendly binary form and operators are fused by whole-stage codegen. Calling .rdd forces a DeserializeToObject step that materialises each row as a real object, which in PySpark additionally pickles it and sends it to a python worker, and the plan shows the codegen island ending right there. Everything after the conversion is opaque to the optimizer: no predicate pushdown into the scan, no column pruning based on later usage, and no Photon or vectorised execution on that branch. Converting back with toDF or createDataFrame restarts the optimizer from a plan whose input is an RDD, so pushdown cannot reach the original files at all. The practical consequence is that one convenience line can turn a pruned, fused scan into a full read plus per-row serialisation in both directions. If you need row-level logic, express it as column functions, or as a pandas_udf when it genuinely must be Python.

df.rdd.map(lambda r: (r.id, r.amount * 2)).toDF(['id', 'amt']).explain()
# DeserializeToObject ... MapPartitions ... no PushedFilters on the source

Common mistake: Dropping to .rdd for one transformation and expecting the rest of the plan to optimise as before.

Follow-up they will ask: How would you express that same map as a column expression?

Lesson · Simulation

advanced

What does mapPartitions give you over map, and what is the bug people hit with it?

RDDs: lineage and partitions · tap to open the answer

Say this first: It hands you an iterator per partition so you can amortise setup cost — and the bug is materialising that iterator into a list.

Remember these

  • mapPartitions runs setup once per task, not once per row
  • list(rows) materialises the whole partition and reintroduces OOM
  • Return a generator or yield so rows stay streamed
  • Resources must stay open until the lazy iterator is consumed

Why: map invokes your function once per record, so anything expensive to construct — a database connection, a tokenizer, a loaded model — is built per row. mapPartitions invokes it once per partition with an iterator over that partition's records, so the setup happens once per task and the teardown can be explicit. The trap is that an iterator is lazy and a list is not: writing list(rows) at the top of the function pulls the whole partition into memory at once, which turns Spark's streaming execution into a per-task memory ceiling and produces exactly the OOM that mapPartitions was supposed to avoid. Return a generator, or yield, so rows stay streamed. Two related mistakes are closing a resource in a way that triggers before the lazy iterator is consumed, and capturing a non-serialisable object in the closure instead of creating it inside the function. The payoff is real when setup is expensive; for pure column arithmetic, a DataFrame expression is still better than either.

def f(rows):
    model = load_model()      # once per partition
    for r in rows:
        yield (r.id, model.score(r))

rdd.mapPartitions(f)

Senior depth: mapPartitions is the right escape hatch exactly when per-partition setup dominates per-row work, and partition count then becomes a tuning knob against setup cost. The failure mode is always memory, because the API hands you something that looks like a collection and is not — if the function body does not read like a stream, it is wrong.

Common mistake: Writing rows = list(rows) for convenience and then reporting that mapPartitions causes executor OOM.

Follow-up they will ask: How would you size partitions for a function whose setup costs two seconds?

Lesson · Simulation

advanced

You wrapped the job in a class and rdd.map now fails with a pickling or 'Task not serializable' error. Explain the mechanism.

RDDs: lineage and partitions · tap to open the answer

Say this first: The closure captured self, so Spark has to serialise the whole enclosing object — including whatever non-serialisable field it holds.

Remember these

  • Referencing self.x in a closure captures the entire enclosing object
  • Bind values to locals so only the value is serialised
  • Broadcast large read-only payloads; build connections in mapPartitions
  • spark.serializer affects data serialisation, not closure capture

Why: A lambda that references self.multiplier does not capture a number; it captures self, and Spark must ship that object to every executor along with the task. If the class holds a SparkSession, a database connection, a logger, or a thread pool, serialisation fails, and in PySpark the message is a PicklingError naming the offending attribute rather than the line you think is at fault. The direct fix is to bind what you need to a local variable before the closure so only that value is captured. For anything large, use a broadcast variable so the payload is shipped once per executor instead of once per task, and for per-partition resources construct them inside mapPartitions so they are never serialised at all. A useful rule is that executor-bound code should be a module-level function or a static method, with configuration passed in as plain values. Kryo via spark.serializer affects how data is serialised, not how closures are captured, so switching it does not fix this class of error.

m = self.multiplier          # capture the value, not self
rdd.map(lambda r: r * m)
b = sc.broadcast(self.big_lookup)  # one copy per executor

Senior depth: Closure capture is the boundary between driver code and executor code, and it is invisible in Python until it fails. I write executor functions as module-level functions taking plain arguments, which makes the boundary explicit and makes the same functions unit-testable without a cluster.

Common mistake: Switching to KryoSerializer to fix a closure-capture error it has nothing to do with.

Follow-up they will ask: When is a broadcast variable the right answer rather than a local capture?

Lesson · Simulation

senior

A legacy source has no DataSource connector and three thousand shards. How do RDDs help, and what do you give up?

RDDs: lineage and partitions · tap to open the answer

Say this first: Parallelise the shard identifiers and fetch inside mapPartitions, then convert to a DataFrame as early as possible.

Remember these

  • Parallelise shard ids, not data; fetch inside mapPartitions
  • Set partition count from executor cores and source rate limits
  • Convert to a DataFrame with an explicit schema immediately
  • You own pushdown, statistics, and retry semantics at that boundary

Why: The pattern is to treat the shard list as the dataset: parallelise the three thousand identifiers into an RDD with a partition count matched to your executor cores and the source's rate limits, then use mapPartitions so one connection serves many shards and rows are yielded as a stream. That gives you exactly what you need from the RDD API — explicit control over parallelism and per-partition resource lifecycle — and nothing more. Convert the result to a DataFrame with an explicit schema immediately, so every downstream operation is back under Catalyst with codegen, pruning, and Photon eligibility. What you give up at the source boundary is pushdown and statistics: the engine cannot ask the legacy system to filter, so any selectivity has to be expressed in the fetch logic by hand. You also own retry semantics, because a failed task re-fetches its shards and the source may not be idempotent, which argues for landing raw into a bronze Delta table and doing all real processing from there. Keep the RDD island as small as you can defend.

shards = spark.sparkContext.parallelize(shard_ids, 200)
rows = shards.mapPartitions(fetch_shards)
bronze = spark.createDataFrame(rows, schema).write.saveAsTable('bronze.legacy')

Senior depth: The discipline is to land raw and process structured. An RDD ingestion island writing straight into bronze Delta keeps the unoptimisable part of the pipeline one stage deep, makes re-processing free, and means a source outage never forces you to re-run the transformations.

Common mistake: Keeping the whole pipeline in RDDs because the ingestion step had to be, instead of converting right after the fetch.

Follow-up they will ask: How would you make the fetch safe against task retries?

Lesson · Simulation