Question 1

We have a nightly notebook that loops over six hundred tables running OPTIMIZE and VACUUM. What does predictive optimization change about that?

Accepted Answer

Databricks runs the maintenance itself, using Unity Catalog's usage and change statistics to decide which tables actually need it rather than touching all six hundred. The cron approach spends the same effort on a table that changed by four rows and a table that was half rewritten, because a loop cannot tell the difference. Predictive optimization uses the statistics Unity Catalog already keeps about how tables are written and queried, and schedules OPTIMIZE, VACUUM, and related maintenance where the predicted benefit justifies the cost. You enable it once at the account, catalog, or schema level and child objects inherit it. The operational win is that the nightly window stops being a fixed multi-hour block that grows with table count. The accounting change is that this work runs on serverless compute and shows up as its own usage, instead of hiding inside your ETL cluster's bill. Maintenance is scheduled by predicted benefit, not by a fixed table list. Enabled at account, catalog, or schema level and inherited downward. Runs on serverless compute and bills on its own line. Replaces the fixed nightly window that grew with every new table.

Question 2

You enabled predictive optimization on the whole catalog, but one schema's tables are never maintained and still have a small-files problem. Why?

Accepted Answer

Predictive optimization only covers Unity Catalog managed tables — external tables and anything still in the Hive metastore are skipped. The feature works on tables whose storage lifecycle Databricks owns, which means Unity Catalog managed tables. External tables point at a location you control and may be written by things Databricks cannot see, so it will not rewrite or vacuum their files. Tables still registered in hive_metastore are outside Unity Catalog entirely and are not candidates at all. Run DESCRIBE DETAIL or check the information schema to confirm whether the schema's tables are MANAGED or EXTERNAL before blaming the feature. The fix is usually a migration to managed tables, not a configuration flag — and until that happens, those tables need to keep their old maintenance job. Coverage is Unity Catalog managed tables only. External tables are skipped because Databricks does not own their files. hive_metastore tables are not candidates at all. DESCRIBE DETAIL tells you MANAGED versus EXTERNAL per table.

Question 3

Your manager wants proof that predictive optimization is doing something before approving it for production. What do you pull?

Accepted Answer

The predictive optimization operations history system table, which lists every operation it ran, on which table, and what it cost. system.storage.predictive_optimization_operations_history records the maintenance operations the feature performed — the operation type, the target table, the metrics, and the usage attributed to it. Join or compare it against system.billing.usage to put a currency number next to the DBUs. On the benefit side, pick the tables it touched most and compare query profiles or bytes scanned before and after, because the claim being made is that queries stop degrading, not that they get faster than they ever were. Present both halves: the maintenance DBUs it spent and the maintenance job it replaced. If you only show the new line on the bill you will lose the argument, because the cost it removed was buried inside an ETL cluster nobody itemised. system.storage.predictive_optimization_operations_history lists what ran. Join to system.billing.usage to convert DBUs into money. Compare bytes scanned before and after on the most-touched tables. The old cron's cost was hidden in an ETL cluster, so compare honestly.

Question 4

Predictive optimization has been on for a month and a key dashboard table is still slow. The team says the feature is not working. What is your diagnosis?

Accepted Answer

Maintenance keeps the layout you chose healthy; it does not choose better clustering keys, so a table clustered on the wrong column stays slow no matter how well it is compacted. There are two different problems that both present as a slow table. One is layout decay — too many small files, data no longer sorted by the keys you picked — and that is what predictive optimization fixes by scheduling incremental OPTIMIZE where it pays. The other is a wrong key: the table is clustered or Z-ordered on a column nobody filters on, so even a perfectly compacted table has to read most of its files. Look at the query profile for the ratio of files pruned to files read; if pruning is poor while file sizes are healthy, the keys are the problem. The fix is ALTER TABLE ... CLUSTER BY on the columns query history shows people actually filtering on, or CLUSTER BY AUTO so Databricks adjusts the keys from observed patterns. Predictive optimization then schedules the incremental clustering work for the new keys. Layout decay and a wrong key both look like 'slow table'. Healthy file sizes with poor pruning points at the clustering keys. ALTER TABLE ... CLUSTER BY changes keys; it does not rewrite history. CLUSTER BY AUTO lets Databricks pick keys from observed queries.

Question 5

An auditor asks why one table was optimized a dozen times last week while another was never touched. How does the scheduler decide?

Accepted Answer

It uses Unity Catalog's change and access statistics to estimate the benefit of maintenance against its cost, so churned-and-queried tables win and cold tables are left alone. Unity Catalog already knows how much data was written to each table and how often it is read, because every commit and every query goes through it. Predictive optimization uses those signals to rank candidate operations: a table taking constant small writes and serving many queries has a high expected payoff from compaction, while a table written once and never read has none. That is the whole point — a cron loop would have spent the same DBUs on both. The evidence for the auditor is system.storage.predictive_optimization_operations_history, which shows the operations and metrics per table, cross-referenced with query history to show the read volume that justified them. The honest caveat is that the decision is heuristic and tuned by Databricks, so you influence it by what you write and query, not by a knob. Decisions come from UC write-churn and read-access statistics. High churn plus high read volume means high expected payoff. Cold tables are intentionally skipped to avoid wasted DBUs. The operations history table is the audit record of what ran.

Question 6

Legal requires that deleted customer records are physically removed from storage within seven days. Can you point at predictive optimization and call that done?

Accepted Answer

No — VACUUM only removes files past the table's retention window, so the deletion deadline is set by delta.deletedFileRetentionDuration, not by the fact that VACUUM is automated. Deleting a row in Delta writes a new version that no longer references the old file; the file itself stays until VACUUM removes it, and VACUUM will not remove files newer than the retention period. The default retention is measured in days and is a table property, so if it is longer than your legal deadline, automated VACUUM still will not delete in time. You set delta.deletedFileRetentionDuration on the tables under the requirement, and you accept that time travel and restore beyond that window stop working for them. Deletion vectors add a wrinkle worth saying out loud: a deleted row can remain inside a data file until the file is rewritten, so the record is logically gone but physically present until maintenance rewrites it. The compliance answer is a per-table retention setting plus evidence from the operations history that VACUUM ran, not 'the platform handles it'. VACUUM only removes files older than the table's retention window. delta.deletedFileRetentionDuration is the knob that sets the deadline. Shortening retention kills time travel and RESTORE beyond that window. With deletion vectors, rows persist in files until those files are rewritten.

Question 7

Walk me through decommissioning the existing maintenance jobs after you turn predictive optimization on. What would you not delete on day one?

Accepted Answer

Run both for a few weeks, then retire maintenance only for UC managed tables that show up in the operations history, keeping explicit jobs for external tables and deadline-bound tables. Enable it at the catalog level and leave the existing job running, because deleting the safety net before you have evidence is how you discover coverage gaps during an incident. After a couple of weeks, pull the set of tables that appear in system.storage.predictive_optimization_operations_history and intersect it with the table list your job iterates over. Anything in both can come out of the job. What stays is the external and hive_metastore tables, which were never covered, and any table with a hard timing requirement that an expected-value scheduler will not honour. Then shrink the job to that residual list and watch the bill shift from your ETL cluster's DBUs to the serverless maintenance line. Record the before-and-after so the cost conversation later is about a shift, not a new expense. Overlap both approaches before deleting anything. Retire per-table, using the operations history as the coverage proof. External, hive_metastore, and deadline-bound tables keep explicit jobs. Expect the bill to move lines, not disappear.

Question 8

You own a metastore with forty catalogs and a dozen teams. Where do you enable predictive optimization, and how do you keep the cost attributable?

Accepted Answer

Enable at account level so new catalogs inherit it, override downward only where you have a reason, and attribute the serverless maintenance spend per catalog from the operations history. Inheritance is the whole design lever: enabling at the account means every catalog a team creates next quarter is covered without a ticket, which is the opposite of the per-table cron world. You then override at catalog or schema level for the exceptions — a sandbox catalog of throwaway tables where maintenance is wasted spend, or a catalog of external tables where it does nothing anyway. For attribution, the operations history carries the catalog and table, so you can roll maintenance DBUs up to whatever ownership mapping you keep, rather than leaving it as one unattributable platform line. The governance work that actually pays is pushing teams toward managed tables, since that is what decides coverage. And set the expectation early that maintenance spend scales with write churn, so a team that triples its ingest will see its maintenance line triple too. Enable at account level so new catalogs inherit coverage by default. Override downward for sandbox catalogs and external-table catalogs. Roll operations history up to an ownership mapping for chargeback. Maintenance spend scales with write churn, so set that expectation early.