Auto Loader
Job lists 40 million objects each hour
Symptoms
- No cloudFiles
- Landing bucket is the whole org dump
Auto Loader interview questions
Interactive Databricks interview questions on Auto Loader. Same topic as /learn/databricks/auto-loader. cloudFiles watches a landing zone incrementally — notifications, checkpoint, schema evolution — instead of listing the whole prefix.
Question 1 of 3
beginner
Auto LoaderWhat problem does Auto Loader solve?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat problem does Auto Loader solve?
Auto Loader · tap to open the answer
Short: Incrementally ingest new files without listing the whole bucket every run.
Detailed: cloudFiles source + notification/listing + checkpoint. It is not a replacement for Delta MERGE logic — it lands bronze.
Common mistake: Every job does dbutils.fs.ls on the landing zone.
Follow-up: Where is Auto Loader's progress stored?
intermediateDirectory listing vs file notification mode — when do you need notifications?
Auto Loader · tap to open the answer
Short: Huge landing zones: listing is slow and expensive. Notifications scale.
Detailed: File events (SNS/SQS, Event Grid, Pub/Sub) tell you what landed. Listing is fine for small prefixes. Misconfigured notifications skip files or duplicate if you also copy.
Common mistake: Notification mode without the cloud event pipeline actually wired.
Follow-up: How does schema inference evolve with rescued data?
seniorAuto Loader missed a day of files. How do you debug without reprocessing the lake?
Auto Loader · tap to open the answer
Short: Checkpoint, cloud event backlog, and whether files arrived in a different prefix.
Detailed: Don't reset the checkpoint as step 1 — you'll duplicate bronze. Compare source listing vs bronze _metadata.file_path. Replay a prefix with a bounded backfill stream.
Senior: Idempotent bronze (path as key) or a separate backfill table you merge once.
Common mistake: rm checkpoint and 'just rerun'.
Follow-up: How do you backfill without doubles?