What happens when Spark executes a job?
One action, four-hour wall clock
Symptoms
- Single job in Spark UI
- Stage 2 is 3.5 hours
- Stage 0–1 were minutes
Spark scenarios interview questions
Production failures: symptoms, Spark UI proof, then the change.
Case 1 of 27 · symptoms
Practice by topic
- Spark Overview
- Spark Architecture
- Driver & Executors
- Execution Flow
- Lazy Evaluation (Code → Plan)
- Catalyst to Job
- Job to Stages
- Stages to Tasks
- Shuffle & Result
- DataFrame
- Transformations
- Actions
- Lazy Evaluation
- RDD
- Partitions
- Shuffle
- Joins
- Caching
- Job / Stage / Task
- DAG
- Catalyst
- Tungsten
- Broadcast Join
- Data Skew
- AQE
- Driver OOM
- Executor OOM