DataYatriSpark, seen under the hood
Simulate
databricks
intermediate
Connecting…

Auto Loader

cloudFiles watches a landing zone incrementally — notifications, checkpoint, schema evolution — instead of listing the whole prefix.

Explain it at my level

  1. Arrive
  2. Notify
  3. Ingest
  4. Schema
Watch the canvas:new filenotificationbronze DeltaLive simulation
Object storage landing zoneorders-01.jsonorders-02.jsonorders-03.jsonFile notificationqueue / cloudFilesbronze.orders_rawDelta tablecheckpointfiles already seenFiles arrive all day. Do not list the whole bucket each hour.
Auto Loader is incremental ingest: notify, read new files, checkpoint, commit.