DataYatriSpark, seen under the hood
Simulate

Simulation/Databricks

Educational simulation
Databricks

Auto Loader

New files notify, ingest, checkpoint — no full bucket list.

Watch the canvas:new filenotificationbronze DeltaLive simulation
Object storage landing zoneorders-01.jsonorders-02.jsonorders-03.jsonFile notificationqueue / cloudFilesbronze.orders_rawDelta tablecheckpointfiles already seenFiles arrive all day. Do not list the whole bucket each hour.
Auto Loader is incremental ingest: notify, read new files, checkpoint, commit.