Interview/Spark

Spark interview questions: What is Apache Spark?

Interactive Spark interview questions on What is Apache Spark?. Practice with the matching lesson. Spark is a distributed compute engine. You write a plan; a driver turns it into tasks that executors run on partitions.

Lesson · Simulation

What is Apache Spark, in one sentence a hiring manager wants?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

What is Apache Spark?

Notebook 'hangs' until the last cell

Symptoms

  • Cells 1–8 return instantly
  • Cell 9 (show) runs 40 minutes

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What is Apache Spark, in one sentence a hiring manager wants?

What is Apache Spark? · tap to open the answer

Say this first: A distributed engine that processes large datasets in parallel across a cluster.

Why: Spark splits data into partitions, runs a task per partition on executors, and only ships results when an action runs. It is not a database and it does not store your lake.

Common mistake: Calling Spark a 'database' or saying it always holds data in memory.

Follow-up they will ask: What is lazy about a DataFrame transformation?

Lesson · Simulation

intermediate

Why can a 20-line notebook do nothing for minutes, then explode when you call show()?

What is Apache Spark? · tap to open the answer

Say this first: Transformations build a plan. Actions execute it.

Why: groupBy / filter / join are lazy. Spark waits for show, count, write, collect. That is when jobs, stages, and tasks appear in Spark UI.

Common mistake: Thinking Spark is slow at parse time because the notebook looks busy.

Follow-up they will ask: Name three actions that trigger a job.

Lesson · Simulation

senior

A job is 'Spark' but the SLA miss is storage. How do you tell?

What is Apache Spark? · tap to open the answer

Say this first: If input is tiny files or a full scan, the cluster is waiting on the lake, not the CPU.

Why: Check Spark UI: many tiny tasks, huge task overhead, or a scan that ignores partition filters. Scaling executors will not fix a thousand 2 MB files.

Senior depth: Fix the files or the plan. Then scale. Never the reverse as a first move.

Common mistake: Adding workers before looking at file sizes and partition pruning.

Follow-up they will ask: What would you change first — cluster size or the table layout?

Lesson · Simulation

beginner

A teammate says 'Spark is faster than MapReduce because it keeps everything in memory.' What is wrong with that sentence?

What is Apache Spark? · tap to open the answer

Say this first: Spark avoids writing intermediate results to the cluster filesystem between steps — it does not hold your dataset in RAM.

Remember these

  • MapReduce wrote to HDFS between steps; Spark pipelines inside a stage
  • Shuffle map output always lands on executor local disk
  • Working memory is spark.memory.fraction; overflow spills to disk
  • Caching is opt-in with persist(), not the default behaviour

Why: MapReduce materialised the output of every map and every reduce to HDFS, so a five-step job paid five durable round trips. Spark pipelines narrow operators inside a single stage and fuses them into one generated Java method, so rows pass through filters and projections without a write in between. Memory still matters, but only as working space governed by spark.memory.fraction: when a sort or aggregation does not fit, the task spills sorted runs to the executor's local disk and you see it in the Spill (Memory) and Spill (Disk) columns on the Stages page. Shuffle output is always written to local disk by the map tasks, in every Spark version — there is no in-memory shuffle. Caching a dataset in RAM is opt-in via persist(), and it is a separate decision from how the engine executes. So the honest sentence is 'fewer durable writes and operator fusion', not 'in memory'.

from pyspark import StorageLevel
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count()  # materialises the cache; check the Storage tab

Common mistake: Repeating the marketing line '100x faster, in-memory' and then being unable to explain the Spill (Disk) column.

Follow-up they will ask: If memory is not the whole story, what does whole-stage codegen actually fuse?

Lesson · Simulation

intermediate

Your manager asks why a 40 GB daily aggregation needs Spark instead of pandas on one big VM. Argue both sides.

What is Apache Spark? · tap to open the answer

Say this first: Spark wins on spill, parallel scan, and recovery; a single large VM often wins on latency and cost at 40 GB.

Remember these

  • Parquet expands 3-5x in memory, so one-process sizing targets the worst day
  • Spark gives partitioned scan, disk spill, and per-task retry
  • Spark charges startup, a driver JVM, and a shuffle per groupBy
  • Default shuffle partition count makes small jobs scheduler-bound

Why: pandas loads the whole frame into one process, and the in-memory footprint of a 40 GB Parquet file is routinely three to five times the file size once strings and indexes are expanded, so the VM has to be sized for the worst day, not the average one. Spark reads the same table as many partitions sized by spark.sql.files.maxPartitionBytes, runs one task per partition, and spills to local disk instead of dying when a partition does not fit. It also retries a failed task up to spark.task.maxFailures instead of losing the whole run. Against that, Spark charges you cluster startup, a driver JVM, and a shuffle for every groupBy — and the default spark.sql.shuffle.partitions will cut a small aggregation into tiny tasks whose scheduling delay dominates their work, visible as a Duration histogram piled near zero on the Stages page. At 40 GB the deciding factors are usually growth rate and whether the output has to feed a lake table that other jobs read. Pick Spark when the data or the fan-out is growing; pick the VM when the job is stable and single-reader.

Common mistake: Answering 'Spark scales' with no number, and never mentioning that 200 shuffle partitions on 40 GB is mostly overhead.

Follow-up they will ask: What would you set spark.sql.shuffle.partitions to for this job, and how would you arrive at the number?

Lesson · Simulation

advanced

You run spark.read.parquet(path) with no action afterwards, and a job still appears in the Jobs tab. Explain it.

What is Apache Spark? · tap to open the answer

Say this first: Reading Parquet needs a schema and a file list, and both are resolved eagerly on the driver — the listing itself can be parallelised into a real job.

Remember these

  • Schema resolution happens at read time, not at action time
  • Large path counts trigger a distributed listing job
  • mergeSchema reads every footer instead of one
  • An explicit schema or a Delta log skips the listing cost

Why: A DataFrame needs a resolved schema before Catalyst can analyse anything, so the Parquet source reads at least one file footer at read time rather than at action time. Path discovery is also eager: the driver lists the directory, and once the number of input paths crosses spark.sql.sources.parallelPartitionDiscovery.threshold, Spark distributes the listing to executors, which shows up as a short job usually labelled with 'parquet at NativeMethodAccessorImpl' or 'InMemoryFileIndex'. On object storage this is the stage that hurts, because listing a prefix with hundreds of thousands of keys is many paginated API calls, not a directory read. If you set spark.sql.parquet.mergeSchema, Spark reads every footer instead of one, which turns a metadata peek into a full-table metadata scan. Supplying an explicit schema, or reading a Delta table whose transaction log already holds the file list and the schema, removes this entirely. So 'transformations are lazy' is true about row processing and false about schema and path resolution.

schema = spark.table('bronze.orders').schema
df = spark.read.schema(schema).parquet('s3://lake/orders')
# no footer read, no listing job

Senior depth: Treat file listing as a first-class cost line on object storage. Delta and Iceberg exist partly because listing is the scaling wall, not the scan. If you must stay on plain Parquet, keep directories shallow, partition coarsely, and never leave mergeSchema on in a scheduled job.

Common mistake: Insisting nothing touches storage until the first action, then having no explanation for a job that appears before one.

Follow-up they will ask: Which file format removes the listing cost entirely, and why?

Lesson · Simulation

senior

The cluster has been up 25 minutes and the Jobs tab is still empty. Give three different explanations and how you would separate them.

What is Apache Spark? · tap to open the answer

Say this first: Either no action was called, the driver is still in planning or file listing, or no executors were ever granted.

Remember these

  • Executors tab showing only the driver means allocation failed
  • Idle executors plus a busy driver means planning or listing
  • Driver thread dump names the blocking stack immediately
  • Empty Jobs tab with healthy everything often means no action

Why: Start with the Executors tab: if the only row is the driver, the cluster manager never gave you containers, and the problem is quota, instance availability, or an init script failing on every worker — not Spark. If executors are registered and idle, the driver is doing driver-only work: catalog resolution, a view that unions hundreds of tables, or an object-store listing against a huge prefix. A driver thread dump settles that in one click, because you will see threads parked in InMemoryFileIndex, the metastore client, or Catalyst rule application. The third case is the boring one: the code built a plan and never called an action, so there is genuinely nothing to run; explain() confirms a valid plan with zero jobs. Each branch has a different owner, which is the point of asking — platform, data layout, or the notebook. Checking executor logs first is wasted time in two of the three cases.

Senior depth: Order the checks by blast radius, not by familiarity: allocation, then driver, then code. Driver thread dump before log grep is the habit that separates people who fix this in two minutes from people who restart the cluster and hope.

Common mistake: Tailing executor stderr for a query that has not submitted a single task.

Follow-up they will ask: Which of those three would autoscaling make worse?

Lesson · Simulation

architect

You inherit a platform where every pipeline is Spark, including dozens that process under a gigabyte. What do you change, and how do you justify it?

What is Apache Spark? · tap to open the answer

Say this first: Route small, predictable jobs off Spark or onto shared serverless compute, and keep Spark for jobs whose data or fan-out actually needs a cluster.

Remember these

  • Sub-GB jobs are dominated by startup and scheduling, not work
  • Tuning shuffle partitions and shared compute beats rewriting first
  • Keep small jobs in Spark when they sit inside a larger lineage
  • Write the threshold down so it is not relitigated per team

Why: A sub-gigabyte job on a dedicated cluster pays cluster startup, a driver JVM, and a shuffle whose partition count was never tuned, so the wall clock is dominated by fixed cost rather than work; the Stages page shows thousands of tasks finishing in tens of milliseconds with visible Scheduler Delay. The cheap first move is not a rewrite: set a sane spark.sql.shuffle.partitions per job, leave AQE coalescing on so small post-shuffle partitions merge, and move these jobs onto shared or serverless compute so they stop paying individual cluster spin-up. Then classify by data volume and dependency shape. Jobs that are small and self-contained belong in a warehouse query or a single-container script; jobs that are small but sit in the middle of a Spark DAG should stay in Spark so you do not fragment the lineage. Publish the rule as a threshold in the platform docs rather than deciding case by case, otherwise every team relitigates it. The honest cost is two execution stacks to operate instead of one.

Senior depth: Instrument it: ratio of cluster-up time to task time per job. Anything above roughly one is paying more for compute existence than for compute. That single ratio turns a taste argument into a backlog you can rank.

Common mistake: Proposing a mass rewrite off Spark before measuring how much of the runtime is cluster startup.

Follow-up they will ask: What metric would you put on a dashboard to find these jobs automatically?

Lesson · Simulation