spark/fundamentals
beginner
Connecting…DataFrames and schemas
A DataFrame is a distributed table with a schema. Catalyst can optimize it; an RDD of tuples cannot.
Explain it at my level
- Schema
- Catalyst
- Physical plan
Step 1 of 4
- 1. Unresolved logicalunresolved attributes
- 2. Analyzed logicalcatalog + types bound
- 3. Optimized logicalpredicate pushdown, column prune
- 4. Physical planFileScan → Exchange → HashAggregate