What is Apache Spark?
Notebook 'hangs' until the last cell
Symptoms
- Cells 1–8 return instantly
- Cell 9 (show) runs 40 minutes
What is Apache Spark? interview questions
Interactive Spark interview questions on What is Apache Spark?. Same topic as /learn/spark/overview. Spark is a distributed compute engine. You write a plan; a driver turns it into tasks that executors run on partitions.
Question 1 of 3
beginner
What is Apache Spark?What is Apache Spark, in one sentence a hiring manager wants?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is Apache Spark, in one sentence a hiring manager wants?
What is Apache Spark? · tap to open the answer
Short: A distributed engine that processes large datasets in parallel across a cluster.
Detailed: Spark splits data into partitions, runs a task per partition on executors, and only ships results when an action runs. It is not a database and it does not store your lake.
Common mistake: Calling Spark a 'database' or saying it always holds data in memory.
Follow-up: What is lazy about a DataFrame transformation?
intermediateWhy can a 20-line notebook do nothing for minutes, then explode when you call show()?
What is Apache Spark? · tap to open the answer
Short: Transformations build a plan. Actions execute it.
Detailed: groupBy / filter / join are lazy. Spark waits for show, count, write, collect. That is when jobs, stages, and tasks appear in Spark UI.
Common mistake: Thinking Spark is slow at parse time because the notebook looks busy.
Follow-up: Name three actions that trigger a job.
seniorA job is 'Spark' but the SLA miss is storage. How do you tell?
What is Apache Spark? · tap to open the answer
Short: If input is tiny files or a full scan, the cluster is waiting on the lake, not the CPU.
Detailed: Check Spark UI: many tiny tasks, huge task overhead, or a scan that ignores partition filters. Scaling executors will not fix a thousand 2 MB files.
Senior: Fix the files or the plan. Then scale. Never the reverse as a first move.
Common mistake: Adding workers before looking at file sizes and partition pruning.
Follow-up: What would you change first — cluster size or the table layout?