What happens when Spark executes a job?
One action, four-hour wall clock
Symptoms
- Single job in Spark UI
- Stage 2 is 3.5 hours
- Stage 0–1 were minutes
Spark scenarios interview questions
Production failures: symptoms, Spark UI proof, then the change.
Case 1 of 36 · symptoms
Practice by topic
- Spark Overview
- Spark Architecture
- Driver & Executors
- Execution Flow
- DataFrame
- Spark SQL
- Transformations
- Actions
- Lazy Evaluation
- RDD
- Partitions
- Repartition vs Coalesce
- Shuffle
- Joins
- Caching
- File Formats
- Window Functions
- UDFs
- Job / Stage / Task
- DAG
- Catalyst
- Tungsten
- Memory Management
- Structured Streaming
- Broadcast Join
- Data Skew
- AQE
- Driver OOM
- Executor OOM