Beginner

Short: A chunk of data that Spark processes in parallel.

Detailed: Spark splits datasets into partitions so tasks can run concurrently across executors. More partitions can increase parallelism, but too many create scheduling overhead.

Senior: Partitioning strategy affects shuffle cost, file layout, and skew. Choose partition counts based on data size, core count, and downstream consumers.

Common mistake: Confusing partitions with tasks or executors.

Follow-up: How would you choose a partition count for a 200 GB dataset?

Short: Records with the same key must move to the same reducer.

Detailed: Aggregation requires co-locating equal keys. Spark inserts an Exchange (hash partitioning) which creates a stage boundary and network transfer.

Common mistake: Assuming groupBy is always a narrow transformation.

Follow-up: When can aggregation avoid a full shuffle?

Short: A distributed engine for large-scale data processing.

Detailed: Spark runs tasks in parallel across executors. You write DataFrame or SQL code; the driver plans and schedules it.

Common mistake: Calling Spark a database.

Follow-up: How is Spark different from a data warehouse?

Short: Driver plans; executors run tasks.

Detailed: The driver holds the SparkContext/SparkSession, builds plans, and coordinates. Executors execute tasks and cache data.

Common mistake: Putting heavy collect() on the driver.

Follow-up: What happens if the driver OOM's?

Short: Driver, cluster manager, executors.

Detailed: The driver plans, the cluster manager allocates, executors compute.

Common mistake: Forgetting the cluster manager.

Follow-up: Where does SparkSession live?

Short: The original Spark API.

Detailed: The original Spark API. An RDD is a partitioned collection with lineage so lost partitions can be recomputed.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: A DataFrame is a distributed table with a schema.

Detailed: A DataFrame is a distributed table with a schema. Catalyst can optimize it; an RDD of tuples cannot.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Narrow transformations stay inside a partition.

Detailed: Narrow transformations stay inside a partition. Wide ones shuffle. That difference is the stage boundary.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Transformations build a DAG.

Detailed: Transformations build a DAG. Actions (count, show, collect, write) submit a job.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Spark waits so Catalyst can push filters into scans and skip unused columns.

Detailed: Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: One partition is one chunk of data.

Detailed: One partition is one chunk of data. One task processes one partition in a stage. Size them for 128–256 MB.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: A shuffle moves records so equal keys co-locate.

Detailed: A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Sort-merge shuffles both sides.

Detailed: Sort-merge shuffles both sides. Broadcast copies the small side. The wrong choice shuffles terabytes.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Caching stores computed partitions so later actions do not recompute the DAG.

Detailed: Caching stores computed partitions so later actions do not recompute the DAG. Unpersist when you are done.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: The driver plans.

Detailed: The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: An action creates a job.

Detailed: An action creates a job. Shuffle boundaries split the job into stages. Each partition in a stage is a task.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: The DAG is the graph of operators.

Detailed: The DAG is the graph of operators. Wide edges are shuffles. Reading the DAG is reading the cost.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Catalyst rewrites your query four times.

Detailed: Catalyst rewrites your query four times. Exchange in the physical plan is the shuffle you will pay for.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Tungsten stores rows off-heap and generates JVM bytecode for whole stages so Spark SQL is not a naive iterator of JVM objects.

Detailed: Tungsten stores rows off-heap and generates JVM bytecode for whole stages so Spark SQL is not a naive iterator of JVM objects.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: When one key owns most of the data, one task becomes the stage.

Detailed: When one key owns most of the data, one task becomes the stage. Salt or AQE skew join.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Replicate a small dimension to every executor and skip shuffling the fact table.

Detailed: Replicate a small dimension to every executor and skip shuffling the fact table.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: AQE uses runtime stats to coalesce shuffle partitions, change join strategy, and split skewed partitions.

Detailed: AQE uses runtime stats to coalesce shuffle partitions, change join strategy, and split skewed partitions.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Databricks control plane is the SaaS UI and jobs service.

Detailed: Databricks control plane is the SaaS UI and jobs service. Data plane is compute in your cloud account, next to your data.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: Delta is ACID on object storage: Parquet files plus a transaction log.

Detailed: Delta is ACID on object storage: Parquet files plus a transaction log. Time travel, OPTIMIZE, and MERGE all come from that log.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?

Short: catalog.

Detailed: catalog.schema.table with metastore-level governance, privileges, and lineage — not hive_metastore.default.

Common mistake: Mixing up partitions, tasks, and executors.

Follow-up: What Spark UI tab proves it?