Say this first: row_number breaks ties arbitrarily, rank gives ties the same number and skips the next, dense_rank gives ties the same number and does not skip.
Remember these
- row_number is 1,2,3; rank is 1,1,3; dense_rank is 1,1,2
- row_number with ties needs a deterministic tiebreaker or it is unstable
- Top-N including ties means rank or dense_rank, not row_number
- The rank filter sits above the window, though newer Spark can push a group limit down
Why: With sales of 100, 100, 90, row_number assigns 1, 2, 3 — the two hundreds are separated in whatever order the sort happened to produce, which is not stable across runs because the sort is not total. rank assigns 1, 1, 3, preserving the tie and skipping position 2. dense_rank assigns 1, 1, 2. For 'top 3 including ties' you want rank() <= 3 or dense_rank() <= 3 depending on whether skipped positions should consume slots; for exactly three rows you want row_number, but then you must add a deterministic tiebreaker column to the orderBy or accept nondeterminism. Note also that the filter on the rank is applied above WindowExec, so conceptually the whole partition is still ordered — though recent Spark versions can push a per-group limit beneath the window for rank-style functions, which you may see as a WindowGroupLimit node in the plan.
w = Window.partitionBy("category").orderBy(F.desc("sales"), F.asc("product_id"))
top3 = df.withColumn("rn", F.row_number().over(w)).where("rn <= 3").drop("rn")
Senior depth: Parallelism collapses to eight. partitionBy is the shuffle key, so eight distinct categories means eight non-empty post-shuffle partitions and eight tasks, each holding an eighth of a billion rows to sort. The Stage page shows a handful of enormous tasks with heavy spill. Pre-aggregate to the grain the ranking actually needs — product totals per category — before ranking, so the window runs over thousands of rows instead of a billion.
Common mistake: Using row_number for 'top N' and silently dropping legitimate ties every run.
Follow-up they will ask: The category column has only eight distinct values on a billion-row table. What goes wrong?