Curriculum
All lessons
Every lesson follows the same path: concept, visualization, code, internals, a production failure, and interview questions. No account needed.
Learn Spark
- beginnerFundamentals
Spark Overview
What Spark is, why it exists, and the mental model of driver, executors, and lazy plans.
- beginnerFundamentals
Spark Architecture
Cluster manager, driver JVM, executor JVMs, and how a SparkSession talks to a cluster.
- beginnerFundamentals
Execution Flow
Trace groupBy + show() from lazy code through jobs, stages, tasks, shuffle, and the result.
- intermediateFundamentals
RDD
Resilient Distributed Datasets: partitions, lineage, and when you still need the RDD API.
- beginnerFundamentals
DataFrame
Structured data with a schema. Catalyst can optimize DataFrames in ways RDDs cannot.
- beginnerFundamentals
Transformations
Narrow vs wide transformations, and why nothing runs until an action arrives.
- beginnerFundamentals
Actions
show, count, collect, write — the calls that force Spark to produce a result.
- beginnerFundamentals
Lazy Evaluation
Spark builds a DAG of transformations and waits. That wait is a feature, not a bug.
- beginnerCore Concepts
Partitions
A partition is a chunk of data. One task processes one partition. Size them for healthy runtime.
- intermediateCore Concepts
Shuffle
Records move across the network so matching keys land on the same reducer.
- intermediateCore Concepts
Joins
Sort-merge, shuffle hash, and broadcast. The strategy Spark picks changes the cost by orders of magnitude.
- intermediateCore Concepts
Caching
persist and cache store computed partitions so later actions do not recompute the whole DAG.
Spark Internals
- beginnerExecution
Driver & Executors
The driver plans. Executors run tasks. Never ship a huge collect() back to the driver.
- intermediateExecution
Job / Stage / Task
An action creates a job. Shuffle boundaries split jobs into stages. Stages split into tasks.
- intermediateExecution
DAG
The directed acyclic graph of RDDs / DataFrame operators Spark will execute.
- advancedQuery Engine
Catalyst
Logical plan → analyzed → optimized → physical plan. Exchange nodes mark shuffles.
- advancedQuery Engine
Tungsten
Off-heap memory, whole-stage codegen, and cache-aware layout that make Spark SQL fast.
Performance
- seniorPerformance
Data Skew
One key owns most of the data. One task runs for 40 minutes while 199 finish in 2.
- intermediatePerformance
Broadcast Join
Replicate a small table to every executor and skip the shuffle on the large side.
- advancedPerformance
AQE
Adaptive Query Execution coalesces shuffle partitions, switches join strategy, and handles skew at runtime.
Databricks
- intermediateDatabricks
Databricks Architecture
Control plane vs data plane, workspaces, clusters, and how jobs run on your cloud account.
- intermediateDatabricks
Delta Lake
ACID transactions on object storage: the _delta_log, Parquet data files, time travel, and OPTIMIZE.
- seniorDatabricks
Unity Catalog
Three-level namespace, governed storage, and how permissions attach to catalogs, schemas, and tables.
https://www.datayatri.com