Interview/Spark

Spark interview questions: File formats: why Parquet reads less

Interactive Spark interview questions on File formats: why Parquet reads less. Practice with the matching lesson. Why Parquet reads a fraction of the bytes CSV does: column pruning, row-group stats, and embedded schema.

Lesson · Simulation

Same rows, same cluster: why does the Parquet version of a table read so much faster than the CSV version?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

File formats: why Parquet reads less

Dashboard query got 30x slower after a 'harmless' upstream format change

Symptoms

  • Upstream team switched a landing table from Parquet to JSON
  • Same row count, same cluster, query time up roughly 30x
  • The scan is now most of the wall clock

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

Same rows, same cluster: why does the Parquet version of a table read so much faster than the CSV version?

File formats: why Parquet reads less · tap to open the answer

Say this first: Parquet is columnar, typed, and carries statistics — a CSV scan has to read and parse every byte of every line.

Why: A Parquet scan reads only the columns in your select and can skip entire row groups using min/max statistics. CSV has no schema, no column boundaries, and no stats, so Spark parses the whole file. The plan shows FileScan parquet with a ReadSchema listing just the columns you asked for.

Common mistake: Saying Parquet is faster 'because it's compressed' and stopping there.

Follow-up they will ask: What is still a fair use case for CSV or JSON?

Lesson · Simulation

intermediate

You read a CSV with inferSchema=true and a job appears in Spark UI before your action ever runs. What is it?

File formats: why Parquet reads less · tap to open the answer

Say this first: Schema inference is an eager extra pass over the data to guess column types.

Why: inferSchema reads the file (all of it for CSV by default) to decide types, which submits its own job ahead of your transformation. Passing an explicit StructType removes that pass; Parquet and ORC never need it because the schema is in the file footer.

Common mistake: Blaming that first slow cell on cluster warm-up instead of the inference job.

Follow-up they will ask: What else can inferSchema get wrong besides being slow?

Lesson · Simulation

senior

Your predicate is on a non-partition column and the job still only touches 3% of the files. What made that work, and when does it stop working?

File formats: why Parquet reads less · tap to open the answer

Say this first: Parquet row-group min/max statistics let Spark skip groups whose value range cannot match the predicate.

Why: Every row group stores per-column min/max; with pushdown the reader drops groups outside the predicate range, which shows up as a small Input Size / Records relative to the table. It only works when data is physically clustered on that column — if values are scattered, every row group's min/max spans the whole domain and nothing is pruned.

Senior depth: Sort or cluster on the filter column at write time (Z-ORDER or liquid clustering on Delta). Statistics are only as useful as the physical layout — this is a layout decision, not a config flag.

Common mistake: Calling row-group skipping 'an index' and expecting it to work on randomly ordered data.

Follow-up they will ask: How do you make the data cooperate?

Lesson · Simulation

architect

The team wants gzip because it compresses the smallest. What do you tell them?

File formats: why Parquet reads less · tap to open the answer

Say this first: A gzip text file is not splittable, so one file becomes one task no matter how large it is.

Why: Gzip must be decompressed from the beginning, so Spark cannot split a gzip CSV or JSON file — a 20 GB file is a single task and the rest of the cluster idles. Snappy and zstd inside Parquet are applied per page and row group, so the file stays splittable; zstd trades some CPU for noticeably smaller files than snappy.

Senior depth: It is the opposite failure: thousands of tiny Parquet files make per-file open and metadata cost dominate, with the task duration histogram piled near zero. Compact on write or with OPTIMIZE and aim for files in the hundreds of MB.

Common mistake: Comparing codecs only on compressed size and ignoring splittability and decode cost.

Follow-up they will ask: Where does the small-file problem fit into this conversation?

Lesson · Simulation

beginner

Describe the physical layout of a Parquet file from the top down.

File formats: why Parquet reads less · tap to open the answer

Say this first: A file is a sequence of row groups; each row group holds one contiguous chunk per column, split into pages, with a footer carrying the schema, offsets, and statistics.

Remember these

  • Row group, then column chunk, then page — that is the nesting
  • The footer carries schema, chunk offsets, and per-chunk min/max/null stats
  • Column pruning is a ranged read of only the chunks you asked for
  • Row-group skipping uses footer statistics, not an index

Why: At the top a Parquet file is divided into row groups, each covering a horizontal slice of rows. Inside a row group each column is stored as its own contiguous column chunk, and each chunk is divided into pages that carry the encoded values plus definition and repetition levels for nulls and nested fields. The footer at the end of the file holds the schema and, for every column chunk, its byte offset and statistics such as min, max, and null count. This is why a reader opens the footer first: it learns the schema with no inference pass, issues ranged reads for only the column chunks in ReadSchema, and drops whole row groups whose min/max cannot satisfy the pushed filter. Grouping like values together is also what makes dictionary encoding and run-length encoding effective before the block codec runs.

Common mistake: Describing Parquet as 'columns stored separately' without row groups, which makes splittability impossible to explain.

Follow-up they will ask: Given that layout, why is a Parquet file splittable but a gzipped CSV is not?

Lesson · Simulation

intermediate

A read with mergeSchema=true sits for eleven minutes before a single row is processed. What is it doing?

File formats: why Parquet reads less · tap to open the answer

Say this first: It is opening the footer of every file in the table to union their schemas before planning can finish.

Remember these

  • mergeSchema reads every file footer, not a sample
  • Cost scales with file count, which is why small files make it unbearable
  • An explicit StructType removes the resolution step entirely
  • Delta and Iceberg keep the schema in the log, so it is one metadata read

Why: Without mergeSchema, Spark reads one file's footer (or the summary metadata) and assumes the rest match, so planning is cheap. With mergeSchema=true it must read every footer and reconcile the field sets, which on a table of hundreds of thousands of objects is hundreds of thousands of small remote reads before any job you care about starts. You see it as a long pause in the driver, or as a schema-inference job in the Jobs tab that precedes your real work. The fix is to stop asking the file layer to resolve schema at all: pass an explicit StructType, or move to a table format like Delta or Iceberg where the current schema lives in the log and resolution is a single metadata read. Keep mergeSchema as a one-off tool for investigating a messy landing zone, not as a standing option on a production read.

from pyspark.sql.types import StructType, StructField, StringType, DoubleType
schema = StructType([StructField("id", StringType()), StructField("amount", DoubleType())])
df = spark.read.schema(schema).parquet("/mnt/raw/events")

Common mistake: Leaving mergeSchema on permanently 'in case the schema changes' and paying for it on every single run.

Follow-up they will ask: What happens when two files genuinely disagree on a column's type?

Lesson · Simulation

advanced

Your filter is where(year(ts) = 2026) and PushedFilters comes back empty. Why, and what do you write instead?

File formats: why Parquet reads less · tap to open the answer

Say this first: A predicate on a function of a column is not a column-to-literal comparison, so it cannot be pushed into the file reader.

Remember these

  • Pushdown needs a bare column compared to a literal
  • year(), cast(), upper() around the column all block it
  • Rewrite as a half-open range on the raw column
  • PushedFilters and PartitionFilters in explain('formatted') are the proof

Why: Filter pushdown translates a Catalyst predicate into a format-level filter, and the Parquet reader can only evaluate simple comparisons between a column reference and a literal against the footer statistics. year(ts) = 2026 wraps the column in an expression, so there is nothing to compare to the min/max of ts and the predicate stays as a Filter node above the scan — every row group is read, decoded, and then discarded. Rewriting it as a half-open range on the raw column, ts >= '2026-01-01' and ts < '2027-01-01', gives the reader exactly the comparison it can use, and Input Size / Records on the FileScan drops accordingly. The same trap catches cast() on a partition column, upper() on a string key, and any predicate buried inside a UDF. Confirm with explain('formatted') and read the PushedFilters and PartitionFilters lines directly rather than assuming.

df.where("year(ts) = 2026").explain("formatted")                      # PushedFilters: []
df.where("ts >= '2026-01-01' AND ts < '2027-01-01'").explain("formatted")  # pushed

Senior depth: Physical clustering. Row-group skipping works by comparing the predicate against each group's min and max, so if the values are scattered across the file every group's range spans the whole domain and nothing can be skipped. Sorting on write, Z-ORDER, or liquid clustering on the filter column is what makes the statistics selective. Statistics are only as good as the layout they describe.

Common mistake: Assuming the optimizer rewrites year(ts) = 2026 into a range automatically for every type and source.

Follow-up they will ask: Your predicate is pushed but Input Size barely moved. What else is required?

Lesson · Simulation

senior

Give me the decision rule for Parquet versus ORC versus Avro — not a feature list.

File formats: why Parquet reads less · tap to open the answer

Say this first: Columnar for analytical reads of a few columns, row-based for whole-record writes and transport; everything else is ecosystem preference.

Remember these

  • Read pattern decides: few columns of many rows means columnar
  • Avro is row-based, which suits per-record writes and transport
  • ORC and Parquet differ mostly in ecosystem, not in capability class
  • The real mistake is letting the ingestion format become the analytics format

Why: The rule is the access pattern. Analytical queries touch a few columns of many rows, so a columnar layout — Parquet or ORC — lets the reader fetch only the needed column chunks and skip row groups on statistics. Ingestion, message transport, and replay touch whole records one at a time, which is where a row-based format like Avro wins: appending one record does not require buffering a row group, and Avro's schema-with-the-data model fits a Kafka pipeline's schema-evolution story. Between the two columnar options the technical differences are real but secondary — ORC carries stride-level indexes and optional bloom filters and is native to the Hive lineage, while Parquet has broader engine and tooling support — so the honest answer is to pick the one your table format and query engines are built around and stop arguing. The failure mode to call out is using a row format for the analytics layer because that is what ingestion produced.

Senior depth: One level above. Delta and Iceberg store data as Parquet or ORC and add a transaction log on top, which gives atomic commits, time travel, schema evolution, and file-level statistics for pruning without listing. So the question is not Delta versus Parquet — it is which file format Delta writes, and whether you want a log above it. For anything a pipeline writes more than once, you want the log.

Common mistake: Reciting benchmark comparisons instead of naming the access pattern that makes the choice.

Follow-up they will ask: Where does a table format like Delta or Iceberg sit in this comparison?

Lesson · Simulation

architect

A 3 TB table is stored in roughly 40 KB files and every consumer is complaining. Design the fix and the guardrail that stops it recurring.

File formats: why Parquet reads less · tap to open the answer

Say this first: Compact to files in the hundreds of MB, fix the writer that produced fragments, and monitor average file size as an SLO.

Remember these

  • Tens of millions of objects means listing and task launch dominate real work
  • Compaction fixes today; the writer is what fixes tomorrow
  • Trigger interval and partition-column cardinality are the usual root causes
  • Track average file size per table as a monitored SLO

Why: At 40 KB per file a 3 TB table is tens of millions of objects, and every read pays listing, footer opens, and task launch per object — the work is all overhead and almost none of it is data. The immediate remedy is compaction: OPTIMIZE on Delta, or a rewrite job that reads and writes back with a sane partition count. That is a one-time fix, so the durable part is the writer: match the write's partition count to the volume per batch, bound file size with maxRecordsPerFile, lengthen the streaming trigger interval so each micro-batch emits fewer files, and avoid partitioning on a column with enough cardinality to guarantee tiny directories. The guardrail is a metric: record file count and average file size per table after every write and alert when the average drops below a threshold, because this problem always arrives gradually and is only noticed when a consumer's SLA breaks.

-- one-time remediation
OPTIMIZE curated.events;
-- and in the writer
SET spark.databricks.delta.optimizeWrite.enabled = true;

Senior depth: Make it incremental and bounded. Compact only the partitions written since the last run rather than the whole table, run it on its own small cluster off the critical path, and cap how much it rewrites per execution so it never becomes a surprise multi-hour job. Full-table compaction on a shared cluster during business hours is how a fix becomes the next incident.

Common mistake: Scheduling compaction and calling it done, while the writer keeps producing fragments every hour.

Follow-up they will ask: Compaction competes with your pipelines for the same cluster. How do you schedule it?

Lesson · Simulation