DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Stages to Tasks — partitions, tasks, executors interview questions

Interactive Spark interview questions on Stages to Tasks — partitions, tasks, executors. Same topic as /learn/spark/stages-to-tasks. Four input partitions become four tasks. The driver schedules those tasks onto executor JVMs — three different nouns.

Lesson · Simulation

beginner
Stages to Tasks — partitions, tasks, executors

What is the difference between a partition and a task?

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Stages to Tasks — partitions, tasks, executors

200 ms of work, 12 minutes of scheduling

Symptoms

  • Thousands of tasks
  • Each reads a tiny file
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

What is the difference between a partition and a task?

Stages to Tasks — partitions, tasks, executors · tap to open the answer

Short: A partition is a slice of data. A task is the unit of work that reads that slice in a stage.

Detailed: One partition → one task in that stage. Executors run many tasks over time. Do not call an executor a task.

Common mistake: Using partition, task, and executor as synonyms.

Follow-up: If you have 200 partitions and 10 executors, how many tasks in a narrow stage?

Lesson · Simulation

intermediate

Why can 10 000 partitions make a small table slower?

Stages to Tasks — partitions, tasks, executors · tap to open the answer

Short: Task scheduling overhead dwarfs the work. Each task has launch cost.

Detailed: Tiny files / too many partitions → thousands of 20 ms tasks. Spark UI shows high scheduler delay. Coalesce or compact files.

Common mistake: repartition(10000) 'for more parallelism' on a 200 MB table.

Follow-up: How do you pick a partition count for a 200 GB shuffle?

Lesson · Simulation

senior

Tasks fail with FetchFailed. What layer actually broke?

Stages to Tasks — partitions, tasks, executors · tap to open the answer

Short: Shuffle fetch — a reducer could not read a map output, often because an executor died or shuffle service lost files.

Detailed: Not 'Spark is random'. Check executor loss, disk, and whether speculative execution is hiding a bad node. Retry may succeed; repeated FetchFailed means shuffle storage or OOM on the map side.

Senior: On the map-side executor local disk (or shuffle service). If that JVM dies hard, reducers fetch-fail.

Common mistake: Increasing retries without looking at executor logs.

Follow-up: Where are shuffle files stored?

Lesson · Simulation