Senior

Short: Compare task metrics in Spark UI for skew, spill, and shuffle.

Detailed: Inspect stage DAG, task duration distribution, shuffle read/write, spill, GC time, and input size. Correlate outliers with key distribution and executor health.

Senior: Build a systematic checklist — skew, small files, wrong join strategy, insufficient parallelism, disk spill, and cluster saturation — and verify with UI evidence before changing configs.

Common mistake: Blindly increasing executor memory without identifying the bottleneck.

Follow-up: What Spark UI panel do you open first?

Short: Size partitions for healthy task duration, minimize wide transformations, and design keys for even distribution.

Detailed: Target ~128–256 MB tasks, avoid unnecessary shuffles, prefer broadcast when safe, use incremental aggregation, and monitor AQE outcomes. Align Delta layout with query predicates.

Senior: Treat shuffle as a first-class cost center in architecture reviews. Design bronze/silver keys, Z-ORDER/liquid clustering, and SLA-driven parallelism budgets.

Common mistake: Copying default spark.sql.shuffle.partitions=200 for all workloads.

Follow-up: How do you validate the design under peak day volume?

Short: Balance cores, memory, and GC against shuffle.

Detailed: Typical starting point is 4–8 cores and 8–32 GB, then adjust from Spark UI GC, spill, and task time.

Senior: Watch for HDFS/S3 concurrency and GC pauses on huge heaps.

Common mistake: Maxing cores per executor always.

Follow-up: When would you use one-core executors?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?

Short: Start from Spark UI stage metrics, then the plan.

Detailed: Compare task duration, shuffle, spill, and input. Confirm with explain() and data profiling.

Senior: Write the hypothesis, the evidence, then the change — in that order.

Common mistake: Restarting the cluster first.

Follow-up: What would you change in the physical plan?