Databricks interview questions: UniForm and Iceberg: one copy, two formats

Say this first: Delta and Iceberg are both Parquet data files plus a metadata layer, so UniForm can publish Iceberg metadata over the files you already wrote instead of copying anything.

Why: The Parquet in a Delta table and the Parquet in an Iceberg table are the same kind of file; what differs is the metadata that lists which files belong to the current snapshot. UniForm uses that fact: set delta.enableIcebergCompatV2 on the table and Databricks writes Iceberg metadata alongside the _delta_log, pointing at the exact same data files. One physical copy, two readable formats, no conversion step. A nightly export instead doubles storage, doubles the write bill, and guarantees the Iceberg copy is hours behind. It also creates a second table that someone will eventually edit independently.

ALTER TABLE sales.orders SET TBLPROPERTIES(
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
)

Common mistake: Describing UniForm as a conversion that rewrites the table into Iceberg files.

Follow-up they will ask: If there is only one copy of the data, what exactly got added to the storage account?

Say this first: No — Iceberg metadata is generated asynchronously after the Delta commit, so Iceberg readers can legitimately lag the Delta log by one or more commits.

Why: A write to a UniForm table commits to the Delta log first and returns. Generating the matching Iceberg snapshot happens after that commit, as a separate step, so there is a window where Delta readers see version N and Iceberg readers still see version N minus one. On a table written every few minutes, that window is almost always open when someone compares the two side by side. You can see where Iceberg is by reading the table properties Databricks maintains for the conversion — the converted Delta version and the converted timestamp — and comparing them with the latest version in DESCRIBE HISTORY. The correct reaction is to treat Iceberg reads as eventually consistent with the Delta table, not to declare a bug.

DESCRIBE EXTENDED sales.orders;
-- compare delta.universalFormat.icebergCompat.convertedDeltaVersion
-- with the latest version from DESCRIBE HISTORY sales.orders

Common mistake: Assuming the Iceberg view is transactionally identical to the Delta log at every instant.

Follow-up they will ask: How would you word the freshness SLA you give the Snowflake team?

Say this first: Storage grew by the Iceberg metadata files only; the Parquet is shared, so the delta is small and does not scale with row count the way a copy would.

Why: UniForm adds Iceberg manifest and snapshot metadata next to the existing _delta_log. Those files describe which Parquet files are in the snapshot, so their size tracks the number of files and the number of commits, not the number of rows. A copy, by contrast, would have doubled the Parquet bytes, which is where the real money is. The number worth watching instead is metadata churn on a high-frequency writer: every commit produces another Iceberg snapshot, so a table written every minute accumulates metadata quickly and benefits from compaction and from expiring old snapshots. Show the storage breakdown by prefix rather than arguing from theory.

Common mistake: Promising 'zero extra storage' when metadata on a frequently committed table is measurable.

Follow-up they will ask: What would you change if the table is written once a minute by a streaming job?

Say this first: Unity Catalog exposes an Iceberg REST catalog endpoint; Trino loads the table through it and UC vends short-lived storage credentials scoped to that table's files.

Why: Trino is configured with the Iceberg REST catalog connector pointed at the Unity Catalog endpoint, authenticating as a principal UC knows — typically a service principal with a token. When Trino asks for a table, UC checks SELECT on that table and returns the current Iceberg metadata location. For the data files, UC performs credential vending: it hands back a temporary, scoped cloud credential instead of you pasting a long-lived storage key into the Trino config. That is the governance win — access is still decided by UC grants, and revoking SELECT stops the next request rather than requiring a key rotation. The two things that break this in practice are a principal with no USE CATALOG and USE SCHEMA, and network rules on the bucket that the vended credential cannot overcome.

-- Trino catalog properties
connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://<workspace>/api/2.1/unity-catalog/iceberg
iceberg.rest-catalog.security=OAUTH2

Senior depth: The point of the REST catalog plus credential vending is that there is exactly one place where access is decided, and it is not the engine's config file. I would stand that up with a dedicated service principal per external engine, grants only on the specific schemas being published, and no direct path access to the bucket for those principals. Then I would watch system.access.audit for reads coming through the Iceberg endpoint, because the failure mode that actually bites is an engine team quietly falling back to a storage key when a grant is missing, and now your audit trail has a hole in it.

Common mistake: Giving the external engine a long-lived storage account key and calling it integration.

Follow-up they will ask: What happens to an in-flight Trino scan when you revoke SELECT from that principal?

Say this first: UniForm requires column mapping and a minimum Delta protocol version, and enabling those upgrades the table protocol in a way you cannot cleanly undo.

Why: Iceberg identifies columns by field ID, so the Delta table has to use column mapping in name mode before Iceberg metadata can describe it. Turning column mapping on raises the table's reader and writer protocol versions, and a protocol upgrade is recorded in the Delta log as a permanent property of the table. The consequence is that any older client — an external Spark pinned to a stale delta-core, a partner job on an old runtime — can stop being able to read or write the table at all. UniForm also restricts some Delta features on the table, because an Iceberg reader has to be able to interpret the same files. So the pre-flight check is: what else reads this table, on what runtime, and is there a feature on it that UniForm will not allow.

ALTER TABLE sales.orders SET TBLPROPERTIES(
  'delta.columnMapping.mode' = 'name',
  'delta.enableIcebergCompatV2' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
)

Senior depth: I would treat it as a protocol migration, not a property change. First inventory the readers per table from lineage and audit, then enable UniForm on one low-traffic table and have every consumer re-run against it. Roll out in waves by consumer group rather than by schema, because the blast radius follows who reads the table, not where it lives. And I would say out loud in the design review that the protocol upgrade is one-way: the fallback plan is a fresh CTAS into a new table, not an ALTER that puts it back.

Common mistake: Treating the ALTER as a reversible toggle and running it across production tables in one script.

Follow-up they will ask: How would you roll this out across a hundred tables without surprising a downstream reader?

Say this first: Iceberg access to a UniForm table is read-only; Delta remains the single writer, so Flink writes to its own table and you ingest from it.

Why: UniForm publishes Iceberg metadata derived from the Delta log. The Delta log is the source of truth, and the Iceberg snapshot is a projection of it generated afterwards. An Iceberg writer committing its own snapshot would have no way to make the Delta log agree, so the two metadata layers would diverge and one set of readers would see files the other does not. That is why the Iceberg surface is for readers. The workable pattern is to give Flink its own table — its own Iceberg table, or a Delta table it writes through a supported path — and then merge that into the shared table from Databricks, so there is exactly one writer per table and the lineage shows it.

Senior depth: Single-writer-per-table is the rule I would not bend, because divergence between two metadata layers over shared files is the kind of corruption nobody detects for weeks. Practically: Flink writes its own bronze table, Databricks runs a streaming MERGE into the shared silver table, and the shared table is the one with UniForm on it. You lose a few seconds of latency and gain a lineage graph where every table has one owner. If the Flink team pushes back on the hop, I would rather spend the argument there than on reconciling two snapshot histories after the fact.

Common mistake: Promising full bidirectional interoperability because 'it is the same Parquet underneath'.

Follow-up they will ask: How would you design the handoff so the Flink stream still lands within a minute?

Say this first: No — that is the opposite direction: a foreign Iceberg table or catalog federation, where Unity Catalog reads someone else's Iceberg metadata.

Why: UniForm is for data you own and write as Delta that you want external engines to read as Iceberg. Reading a partner's Iceberg table is the inbound case: you register their catalog as a connection in Unity Catalog and the tables appear as foreign tables you can query with normal SQL. The data stays in their storage and their catalog remains authoritative, so you get reads, not ownership. The practical constraints are the ones you would expect from a federated read — write support is limited, performance depends on their file layout, and you are now coupled to their refresh cadence and their credentials. If you need to join it hot against your own gold tables repeatedly, materializing a local copy on a schedule is often the honest answer.

Senior depth: The distinction matters because the direction decides who is on the hook for an incident. With UniForm I own the write path, the layout, and the freshness SLA. With a foreign catalog the partner owns all three and I own only the complaint. So I would federate for discovery and ad hoc analysis, and materialize anything that sits under a dashboard SLA — with the materialization job visible in lineage so nobody mistakes the local copy for the live source.

Common mistake: Using 'UniForm' as a blanket word for any Delta-Iceberg interoperability, in either direction.

Follow-up they will ask: Where would governance live if half your gold layer is foreign tables?

Say this first: One Delta gold layer with UniForm enabled, published through Unity Catalog's Iceberg REST catalog, with per-engine service principals and a documented freshness lag.

Why: The physical design is one set of Parquet files written by Databricks, with Delta as the write format and Iceberg metadata published alongside. Unity Catalog is the single catalog: external engines come through the Iceberg REST endpoint with credential vending rather than storage keys, so grants, lineage, and audit cover every reader. The organizational design matters as much as the technical one: declare which tables are published, hold them to a schema contract, and state the Iceberg lag in the contract so nobody builds a reconciliation job on an assumption of exact parity. Publish a narrow set of gold tables rather than the whole catalog, because every published table is an interface you now have to keep stable. The copies you still accept are deliberate ones — a reader that needs sub-second latency or a format you do not publish.

GRANT USE CATALOG ON CATALOG prod TO `trino-reader`;
GRANT USE SCHEMA ON SCHEMA prod.gold TO `trino-reader`;
GRANT SELECT ON TABLE prod.gold.orders TO `trino-reader`

Senior depth: The architecture question is not 'can these engines read Delta' — UniForm answers that. It is 'how many interfaces am I willing to support'. So I would publish a small, versioned gold surface, make UC the only access path, and refuse requests to expose silver. Then I would instrument it: audit reads per principal, watch which published tables nobody has queried in ninety days, and retire them. The failure mode of multi-engine lakehouses is not storage cost, it is a hundred implicit contracts you cannot change because you do not know who depends on them.

Common mistake: Enabling UniForm catalog-wide and calling it a data product strategy, with no contract or ownership.

Follow-up they will ask: Which engine would you still let keep its own copy, and why?