DataYatriSpark, seen under the hood
Simulate
spark/fundamentals
intermediate
Connecting…

RDDs: lineage and partitions

The original Spark API. An RDD is a partitioned collection with lineage so lost partitions can be recomputed.

Lesson 10 of 29 · Spark path

Explain it at my level

  1. Partition
  2. Lineage
  3. Recompute
Watch the canvas:RDD partitionlineageDataFrame pathLive simulation
RDD lineageHadoopRDDfilesMapPartitionsfilterDataFrame / DatasetCatalyst · TungstenPartitions + lineage. That is an RDD.
RDDs are lineage + partitions. DataFrames sit on top and get the optimizer.