Databricks interview questions: Delta Sharing and Clean Rooms

Say this first: The recipient reads your existing Delta table directly from your storage through the sharing protocol, so there is no copy, no export job, and revocation is immediate.

Why: With a file drop you build an export job, pay to write a second copy, and the partner's data is only as fresh as the last run. Delta Sharing removes the copy: you create a share, add the table, create a recipient, and grant SELECT on the share. When the recipient queries, the sharing server checks their permission and returns short-lived pre-signed URLs, and their client reads the Parquet directly from your object storage. The data they see is the live table, so there is no export schedule to go wrong. Revoking is a single grant change rather than chasing files that were already downloaded. It is an open protocol, so the recipient does not have to be a Databricks customer.

CREATE SHARE partner_metrics;
ALTER SHARE partner_metrics ADD TABLE prod.gold.daily_revenue;
CREATE RECIPIENT acme;
GRANT SELECT ON SHARE partner_metrics TO RECIPIENT acme

Common mistake: Describing Delta Sharing as a managed replication service that pushes data to the recipient.

Follow-up they will ask: What does the recipient's client actually download, and from where?

Say this first: Yes — Databricks-to-Databricks sharing uses the recipient's Unity Catalog sharing identifier and mounts as a catalog; open sharing issues a credential file for any client.

Why: For a Databricks recipient you create the recipient with their metastore's sharing identifier, which establishes a trusted relationship between the two Unity Catalogs. On their side the share appears as a catalog they can query with normal three-level names, and their reads carry their own identities, which makes auditing and onward governance much better. For a non-Databricks recipient you use open sharing: Databricks generates an activation link that yields a credential file containing a bearer token and endpoint, and any Delta Sharing client — the Python connector, pandas, Spark, a BI tool — can use it. The tradeoff is that the credential file is a secret you have handed out: anyone who holds it is the recipient, so it needs rotation and an expiry, and you lose per-user attribution on their side.

-- Databricks-to-Databricks
CREATE RECIPIENT acme USING ID 'aws:us-west-2:<metastore-uuid>';
-- open sharing
CREATE RECIPIENT ext_team;
ALTER RECIPIENT ext_team SET PROPERTIES ('expiration_time' = '2026-12-31')

Common mistake: Using open sharing for a Databricks recipient out of habit and giving away a long-lived bearer token.

Follow-up they will ask: How would you rotate an open-sharing credential without breaking their jobs?

Say this first: Revoking the grant stops the next metadata or URL request immediately, but pre-signed URLs already issued remain valid until they expire.

Why: Access is checked when the recipient asks the sharing server for a table's files, and the server responds with pre-signed URLs that are valid for a limited time. Revoke SELECT on the share and the next request is refused, so nothing new can be fetched. A query that was already in flight holds URLs that will keep working until they expire, which is a short window but not zero. So the honest answer is 'effective immediately for new access, with a bounded tail measured in the URL lifetime'. You can also delete the recipient entirely, which invalidates their credentials rather than just one share. What you cannot undo is anything they already read and stored on their side, which is a contractual control, not a technical one.

REVOKE SELECT ON SHARE partner_metrics FROM RECIPIENT acme;
DROP RECIPIENT acme

Common mistake: Promising instant, total cutoff without mentioning in-flight pre-signed URLs or data the recipient already pulled.

Follow-up they will ask: What would you write into the contract to cover the part technology cannot enforce?

Say this first: The recipient reads directly from your object storage, so their scans are your egress — and a cross-region or cross-cloud recipient pulling full tables pays that at your expense.

Why: The no-copy property means the bytes move from your bucket to their compute, and your cloud provider charges you for that egress. A recipient in the same region and cloud is cheap; one in another region or another cloud is not, and a recipient that re-reads the full history on every run multiplies it. The usual cause is their job doing a full scan because they have no predicate on the partition or clustering columns, or because their orchestration refreshes from scratch nightly instead of reading the increment. Find it by correlating storage egress metrics with the recipient's read pattern in the audit records, then look at what their query is actually scanning. The fixes are cooperative: share a table laid out so their filters prune, share an aggregate instead of raw rows, or ask them to read incrementally through change data feed.

Senior depth: I price sharing as a product, because the cost is real and it lands on the provider by default. That means deciding deliberately what gets shared — a narrow aggregate rather than raw silver, laid out so the recipient's likely filters prune — and writing an expected read volume into the agreement. Then I monitor egress per recipient and treat a step change as a conversation, not a surprise. The alternative, where any recipient can rescan your full history nightly for free, is a cost line nobody owns until it is large enough to be an escalation.

Common mistake: Assuming no-copy means no cost, and forgetting that the provider's storage account is paying the egress.

Follow-up they will ask: How would you charge that egress back to the recipient's contract?

Say this first: Add the table to the share WITH HISTORY and enable change data feed, which lets the recipient read versions or changes — at the cost of retaining history you would otherwise vacuum.

Why: By default a shared table gives the recipient the current snapshot. Sharing WITH HISTORY exposes previous versions, which is what lets their client read a range rather than the whole table, and with change data feed enabled on the table they can consume inserts, updates, and deletes directly. The cost is retention: the files backing those versions must still exist, so your VACUUM retention has to cover the window the recipient relies on, and that is extra storage. It also means the recipient can time travel across that window, which is a disclosure decision — they can see what a row looked like before you corrected it. Set the retention window deliberately and state it in the agreement, rather than discovering the mismatch when their incremental read fails because the files were vacuumed.

ALTER TABLE prod.gold.daily_revenue SET TBLPROPERTIES('delta.enableChangeDataFeed' = 'true');
ALTER SHARE partner_metrics ADD TABLE prod.gold.daily_revenue WITH HISTORY

Senior depth: History sharing turns an internal implementation detail — how long I keep old files — into a contractual obligation, and that is the part teams miss. So I set the table's deleted-file retention explicitly to cover the recipient's agreed window plus a margin, document it, and monitor that the recipient's last-read version stays inside it. I also think about disclosure before enabling it: exposing history means exposing restatements, and there are tables where the previous value of a row is more sensitive than the current one. That decision belongs with the data owner, not with whoever ran the ALTER SHARE.

Common mistake: Enabling history sharing without raising retention, so the recipient's incremental reads break after a vacuum.

Follow-up they will ask: What retention window would you negotiate, and what does it cost in storage?

Say this first: The system.access audit tables record Delta Sharing activity on the provider side, including the recipient, the share, and the object accessed.

Why: Because every read goes through the sharing server for permission and URL issuance, the provider sees each access and it is recorded in the audit tables under the system.access schema. You can group by recipient and shared object to produce exactly the report compliance wants, and you can spot patterns like a recipient pulling the whole table nightly. The limits are worth stating: the audit records recipient-level activity, so with open sharing you see the recipient, not which human on their side ran the query, whereas Databricks-to-Databricks sharing carries more identity detail. Audit tables also need their schema enabled on the metastore and have no backfill, so if this is a contractual requirement, enable it before the first contract rather than after the first audit. Build the report as a scheduled query over the audit tables and keep its output, so you are not dependent on audit retention.

SELECT request_params.recipient_name, request_params.share, COUNT(*) AS reads
FROM system.access.audit
WHERE service_name = 'unityCatalog' AND action_name ILIKE '%deltaSharing%'
  AND event_date >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL

Senior depth: I set this up before signing rather than after, because audit tables do not backfill and 'we will produce it when asked' quietly means 'we cannot produce last quarter'. Practically: enable the audit schema on the metastore, run a scheduled job that snapshots recipient activity into a retained reporting table, and reconcile it monthly against the active contracts. If a partner needs per-user attribution, that is an argument for Databricks-to-Databricks sharing or for per-team recipients rather than one shared credential, and it is cheaper to make that a contract term than a retrofit.

Common mistake: Promising per-end-user attribution for an open-sharing recipient, where you only see the recipient identity.

Follow-up they will ask: How would you get per-user attribution for a partner who insists on it?

Say this first: No — that is a clean room: approved code runs over both parties' inputs and only the agreed aggregate output comes back, with neither side reading the other's rows.

Why: Delta Sharing gives the recipient read access to rows you chose to share, which is exactly what both parties are refusing here. A clean room instead creates a collaboration environment where each side contributes tables, and only notebooks or queries both sides approve may run against them. The output is restricted to what was agreed — typically aggregates above a minimum count — so neither party can read the other's raw identifiers. The real design work is in the rules rather than the plumbing: what join key is permitted, what aggregation floor prevents re-identification of a small cohort, and who can approve a new piece of code. Treat it as a joint governance agreement with compute attached, not as a fancier share.

Senior depth: The question I push on is what the output is allowed to be, because that is where the privacy actually lives — the environment just enforces it. Hashed identifiers are not anonymisation; they are pseudonymisation, and with a shared salt they match perfectly, which is the whole point and also the whole risk. So I would insist on an aggregation floor, a fixed set of approved analyses rather than open query access, and a review step for any new code. That is slower than a share, and it is slower on purpose, because the thing being protected is the one asset neither company can get back after a mistake.

Common mistake: Proposing a share of hashed identifiers and calling it privacy-preserving, when a hashed join key is still a join key.

Follow-up they will ask: What minimum cohort size would you require on the output, and why?

Say this first: A narrow versioned set of shared gold tables, one recipient per partner with expiry, D2D where possible, and per-recipient audit and egress monitoring — and no sharing of silver or raw.

Why: The technical pattern is straightforward: a dedicated publishing schema of contracted gold tables, one share per product and one recipient per partner so revocation and audit are per-partner, with recipient expiration set and credentials rotated for open sharing. The decisions that matter are editorial. You refuse to share silver or raw, because every shared table is an interface whose schema you can no longer change freely, and raw grain is where accidental disclosure lives. You version the shared tables so a breaking change is a new table rather than a surprise, and you state freshness and history retention in the contract because both are now obligations. Operationally you need per-recipient egress and read monitoring, since the provider pays for recipient scans, and a quarterly review that retires shares nobody has read. Without that review, you accumulate forty permanent interfaces and lose the ability to evolve your gold layer.

ALTER RECIPIENT acme SET PROPERTIES ('expiration_time' = '2027-06-30T00:00:00Z');
SELECT * FROM system.information_schema.shares;
SHOW GRANTS ON SHARE partner_metrics

Senior depth: The architectural cost of sharing is not storage or compute, it is schema freedom, and that bill arrives about a year in. So the design I defend is a small published surface with explicit versions and a deprecation process agreed up front: a new version is a new table, the old one has an announced end date, and recipient read activity tells you who still depends on it. I would also keep the publishing schema physically separate from the internal gold layer, even when the content overlaps, so internal evolution is never blocked by an external contract.

Common mistake: Sharing whatever each partner asks for, then being unable to change a gold column because forty contracts depend on it.

Follow-up they will ask: How would you deprecate a shared table that three partners still read?