Three 'revenue' numbers in three dashboards
Symptoms
- Each team aggregated bronze differently
- No gold owner
Interactive Databricks interview questions on Medallion Architecture. Practice with the matching lesson. Bronze keeps the raw files. Silver is typed and joined. Gold is the metric table BI actually queries.
Question 1 of 9
What are bronze, silver, and gold for?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 9 of 9 questions.
What are bronze, silver, and gold for?
Medallion Architecture · tap to open the answer
Say this first: Bronze = raw replay. Silver = conformed truth. Gold = product metrics.
Why: Bronze keeps original bytes. Silver types, dedupes, joins. Gold is the grain BI/ML consume. Arrows only flow downstream.
Common mistake: Medallion as three folders with the same messy table copied three times.
Follow-up they will ask: Who is allowed to write back into bronze?
Why not serve bronze to a dashboard?
Medallion Architecture · tap to open the answer
Say this first: Every filter becomes a full scan of landing JSON, and metrics disagree.
Why: Bronze has no stable schema contract. Gold is small, named, and owned. Silver is what jobs share so gold doesn't re-parse.
Common mistake: Skipping silver 'to move faster'.
Follow-up they will ask: Where do quality expectations live?
How do you handle a late-arriving correction in medallion?
Medallion Architecture · tap to open the answer
Say this first: Land it in bronze, merge into silver on the business key, rebuild gold from silver — never patch gold by hand.
Why: If you edit gold, you can't reconstruct yesterday. MERGE in silver with event time. Gold is a projection.
Senior depth: Drop silver+gold, rebuild from bronze, numbers match. If they don't, gold had hidden logic.
Common mistake: UPDATE gold.daily_revenue in a notebook because finance asked.
Follow-up they will ask: What's the replay test?
Why is bronze append-only? Storage is not free, and we already have the cleaned data in silver.
Medallion Architecture · tap to open the answer
Say this first: Bronze is the replay source — overwrite it and you can never rebuild silver and gold after you find a logic bug.
Remember these
Why: Every downstream bug is fixed by rerunning the transform, and that rerun needs the original bytes, including the rows your current logic rejects or ignores. Append-only bronze also gives you the cheapest form of idempotent ingest, because the source file path is a natural key — _metadata.file_path lets you detect a file that landed twice. Cost is controlled with retention and layout rather than with overwrites: cluster or partition bronze by ingest date, compact it, and expire it on a stated schedule you can defend. Overwriting bronze nightly is how a team ends up asking a vendor to resend three weeks of data, which is usually impossible. The rule of thumb is that bronze is cheap storage buying you expensive optionality.
SELECT _metadata.file_path, count(*)
FROM ops.bronze.orders_raw
GROUP BY 1 HAVING count(*) > 0Common mistake: Overwriting bronze nightly because silver is treated as the real table.
Follow-up they will ask: What do you do with rows that fail to parse at the bronze stage?
Deduplication, currency conversion, and a seven-day rolling average. Which layer does each belong in, and why?
Medallion Architecture · tap to open the answer
Say this first: Dedup and currency conversion in silver, the rolling average in gold.
Remember these
Why: Silver owns the correctness that every consumer needs: one row per business key, typed columns, conformed units and codes, and joins to reference data. Gold owns the shape one named consumer wants: grain, aggregation, and metric definitions. The test that separates them is whether two teams would implement the rule identically — dedup and FX normalisation are objectively right or wrong, so doing them twice is both waste and a guaranteed source of two different numbers. A seven-day rolling average is a product decision with a window someone chose, which is why it needs an owner and belongs in gold. Push the aggregate into silver and you have pre-empted the next consumer who needs a different window; push dedup into gold and every gold table reimplements it slightly differently.
Common mistake: Moving aggregation into silver because a dashboard was slow, then needing a different window a month later.
Follow-up they will ask: Where does a business rule like 'exclude internal test accounts' belong?
Rerunning yesterday's job doubled the gold numbers. Make the pipeline idempotent, layer by layer.
Medallion Architecture · tap to open the answer
Say this first: Bronze keyed on the source file, silver MERGE on the business key, gold rewritten with a bounded overwrite of the window rather than appended.
Remember these
Why: Idempotence means rerunning a window converges on the same state instead of adding to it, and each layer needs a different mechanism to get there. Bronze: ingest by file with Auto Loader's checkpoint, or a MERGE keyed on _metadata.file_path, so the same file cannot land twice. Silver: MERGE on the business key with an event-time or version tiebreaker, so a replayed row updates rather than inserts. Gold: never append — use INSERT OVERWRITE or a replaceWhere overwrite scoped to exactly the window being recomputed, so a rerun replaces the slice it owns and touches nothing else. The last piece is where the window comes from: pass it in as a job parameter, because a notebook calling current_date() recomputes the wrong day the moment a retry crosses midnight. Each layer being individually idempotent is what makes the whole pipeline safe to rerun.
df.write.format("delta").mode("overwrite")
.option("replaceWhere", "event_date = '2026-10-23'")
.saveAsTable("ops.gold.daily_revenue")Senior depth: I treat 'can I rerun this window safely' as an acceptance criterion for every pipeline, because retries and backfills are not exceptional events. That means each layer's write is a converging operation and the window is an explicit input, which also makes the pipeline testable: run the same window twice in staging and assert the row counts are identical. The failure I guard against hardest is gold appends, because they are the one that produces plausible-looking wrong numbers instead of an error.
Common mistake: Making silver idempotent and leaving gold as an append, so every backfill doubles the metric.
Follow-up they will ask: What breaks if the job derives its own date from now()?
The source added a field and renamed another. How does that ripple through bronze, silver, and gold, and where should the job fail?
Medallion Architecture · tap to open the answer
Say this first: Bronze absorbs it with schema evolution and rescued data, silver fails or quarantines because it holds a contract, and gold should never see it.
Remember these
Why: Bronze's job is to lose nothing, so evolution belongs there: a new column appends to the schema and unexpected fields land in a rescued-data column rather than being silently dropped. Silver is where the contract lives, and the rename means the column silver reads is now null — a correctness bug that must fail loudly or route the rows to a quarantine table with a reason, not pass through with defaults. If silver holds the line, gold is a projection of silver and is unaffected, so dashboards show yesterday's numbers rather than a confident zero, which is the far better failure. The observable surfaces are a row-count or null-rate expectation on the silver load, and DESCRIBE HISTORY on bronze showing the schema-change commit next to the upstream release date. The anti-pattern to name is coalescing the missing column to a default so the pipeline stays green while the metric quietly goes to zero.
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaEvolutionMode", "rescue")
.load("abfss://lake@acct.dfs.core.windows.net/raw/orders")Senior depth: I design each layer's reaction to drift explicitly, because schema change is routine and the only question is who absorbs it. Bronze is permissive and lossless, silver is strict and loud, gold is derived and therefore immune. Then the pipeline needs one thing more to be operable: an expectation on the silver load — null rate on contract columns, row count against the previous run — so the alert arrives before a consumer notices. A green pipeline producing zeros is strictly worse than a red one, and that is the principle I design the failure modes around.
Common mistake: Coalescing the renamed column to a default so the pipeline stays green and the metric silently drops to zero.
Follow-up they will ask: What would you alert on so this is caught on the first run rather than in a dashboard?
A team proposes a fourth layer called platinum for ML features. Yes or no, and why?
Medallion Architecture · tap to open the answer
Say this first: Only if it has a named consumer and a genuinely different contract; otherwise it is a gold table and the extra layer is one more copy to keep in sync.
Remember these
Why: Layers are justified by a change in contract — fidelity in bronze, correctness in silver, consumer grain in gold — not by the number of transformations between them. ML features often do have a distinct contract: point-in-time correctness so training sees no future information, a serving path with latency requirements, and a schema owned by the model rather than by a dashboard. That is a legitimate reason to model them separately, usually as feature tables in their own schema with their own lineage. What does not justify a layer is that gold became messy, because that is a modelling problem and a fourth copy makes it worse by adding another thing to backfill whenever the logic changes. The three questions I ask of any proposed layer are who owns it, what its SLA is, and what breaks if we delete it.
Senior depth: I let the contract decide, and I make the proposer state it. If they can describe a guarantee gold does not provide — no leakage from the future, a sub-second serving read, a schema versioned with the model — then it is a real layer and I would rather it be explicit than smuggled into gold. If they cannot, the honest answer is that gold needs restructuring and a new name will not fix it. Layer count is not architecture; the number of contracts you can actually enforce is.
Common mistake: Adding a layer per team so each new consumer creates another copy of the same numbers.
Follow-up they will ask: What point-in-time requirement makes a feature table different from a gold aggregate?
Map medallion onto Unity Catalog and compute for a platform with twelve pipelines. What do you standardise?
Medallion Architecture · tap to open the answer
Say this first: Layers as schemas inside a domain catalog, write access per layer limited to the owning service principal, and different retention, quality, and compute per layer.
Remember these
Why: Make the layer part of the name — sales_prod.bronze.orders_raw, sales_prod.silver.orders, sales_prod.gold.daily_revenue — so a grant can be made on a whole layer and the lineage graph reads like the architecture diagram. Only the pipeline's service principal gets MODIFY on silver and gold; humans get SELECT on gold and nothing on bronze, which eliminates the whole class of incident where someone patched a gold table by hand because finance asked. Retention and PII treatment differ by layer, so masks and row filters attach at silver and gold while bronze stays locked down and expires on a schedule. Compute differs too: job or serverless compute for the pipelines, SQL warehouses for gold consumption, and nobody scheduling production work on all-purpose. The last piece is reconciliation — a row-count or revenue check between silver and gold per run — because three named layers are only a contract if something verifies it.
GRANT SELECT ON SCHEMA sales_prod.gold TO `analysts`;
GRANT MODIFY ON SCHEMA sales_prod.silver TO `sp-sales-pipeline`;Senior depth: The architecture is only real if the permission model enforces it, so I map each layer to a securable and each pipeline to a service principal, then grant narrowly. Once humans have no MODIFY on gold, the hand-patched-metric incident cannot happen; once bronze is read only by the silver pipeline, nobody builds a dashboard on raw JSON. Compute follows the same logic: pipelines on job or serverless compute, consumption on warehouses, all-purpose for humans only. Then I add the reconciliation check per run, because a contract nobody verifies drifts within a quarter.
Common mistake: Implementing layers as folder prefixes in one schema, so grants and lineage cannot tell them apart.
Follow-up they will ask: Who should be allowed to write to bronze, and with what identity?