Scenario Based

Symptoms

• Stage duration dominated by a single task
• Cluster CPU mostly idle near the end of the stage

Likely cause

Data skew on a shuffle key

How to investigate

• Open Spark UI → Stages → identify longest task
• Compare input size and shuffle read per task
• Check key cardinality for the join/groupBy key

Spark UI evidence

• Task duration histogram with a long tail
• One partition far larger than others

Possible solutions

• Salt skewed keys
• Enable AQE skew join handling
• Broadcast the smaller side when possible

Tradeoffs

• Salting increases shuffle write volume
• Broadcast requires enough executor memory

Symptoms

• Application killed
• Java heap space on driver

Likely cause

collect() or toPandas() pulled a large result to the driver

How to investigate

• Find the action in the notebook
• Check result size in Spark UI

Spark UI evidence

• A job whose result size is gigabytes

Possible solutions

• Write to storage instead
• Limit/sample before collect

Tradeoffs

• You lose an interactive Python dataframe

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

The original Spark API. An RDD is a partitioned collection with lineage so lost partitions can be recomputed.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

A DataFrame is a distributed table with a schema. Catalyst can optimize it; an RDD of tuples cannot.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Narrow transformations stay inside a partition. Wide ones shuffle. That difference is the stage boundary.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Transformations build a DAG. Actions (count, show, collect, write) submit a job.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

One partition is one chunk of data. One task processes one partition in a stage. Size them for 128–256 MB.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Sort-merge shuffles both sides. Broadcast copies the small side. The wrong choice shuffles terabytes.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Caching stores computed partitions so later actions do not recompute the DAG. Unpersist when you are done.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

An action creates a job. Shuffle boundaries split the job into stages. Each partition in a stage is a task.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

The DAG is the graph of operators. Wide edges are shuffles. Reading the DAG is reading the cost.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Catalyst rewrites your query four times. Exchange in the physical plan is the shuffle you will pay for.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Tungsten stores rows off-heap and generates JVM bytecode for whole stages so Spark SQL is not a naive iterator of JVM objects.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

When one key owns most of the data, one task becomes the stage. Salt or AQE skew join.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Replicate a small dimension to every executor and skip shuffling the fact table.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

AQE uses runtime stats to coalesce shuffle partitions, change join strategy, and split skewed partitions.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Databricks control plane is the SaaS UI and jobs service. Data plane is compute in your cloud account, next to your data.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

Delta is ACID on object storage: Parquet files plus a transaction log. Time travel, OPTIMIZE, and MERGE all come from that log.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count

Symptoms

• SLA missed
• One stage dominates wall clock

Likely cause

catalog.schema.table with metastore-level governance, privileges, and lineage — not hive_metastore.default.

How to investigate

• Open Spark UI Stages
• Run explain('formatted')
• Profile the key distribution

Spark UI evidence

• Task duration histogram
• Shuffle read skew

Possible solutions

• Fix the plan (broadcast, salt, coalesce)
• Fix the files (compact, Z-ORDER)
• Then scale

Tradeoffs

• Broadcast needs memory
• Salting increases task count