Interview/Databricks

Databricks interview questions: Photon

Interactive Databricks interview questions on Photon. Practice with the matching lesson. Photon is Databricks' native vectorized engine. Eligible operators run off-JVM; UDFs often fall back to Spark.

Lesson · Simulation

beginner
Photon

What is Photon?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Photon

Warehouse on Photon, same 40-minute dashboard

Symptoms

  • UDF for currency conversion
  • Tiny files in gold

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

What is Photon?

Photon · tap to open the answer

Say this first: Databricks' native vectorized engine for eligible SQL/DataFrame operators.

Why: It runs in C++ on the data plane, not a new storage format. If the plan is eligible, scans/joins/aggs get faster. UDFs and some expressions fall back to the JVM.

Common mistake: Photon as a magic switch that always 10×s every notebook.

Follow-up they will ask: Where do you see whether Photon ran?

Lesson · Simulation

intermediate

A Python UDF in the query — what happens to Photon?

Photon · tap to open the answer

Say this first: That operator (and often the pipeline around it) falls back.

Why: Photon is a native vectorized engine that only covers operators it has implemented, and a Python UDF is opaque to it. Rows have to leave the native columnar format, be serialized to a Python worker process, evaluated one at a time, and come back, so that operator runs on the JVM path instead. Because the fallback breaks the native pipeline, neighbouring operators often drop out of Photon too, and you end up paying the Photon DBU premium for a query that is barely using it. The query profile labels which nodes ran in Photon, so check there rather than assuming. Rewriting the UDF with built-in SQL functions is the fix; a pandas UDF reduces the per-row serialization cost but is still not Photon-native.

Common mistake: Turning Photon on and keeping the UDF.

Follow-up they will ask: Does Photon run on all-purpose, jobs, and warehouses?

Lesson · Simulation

senior

Photon on, runtime unchanged. How do you investigate?

Photon · tap to open the answer

Say this first: The bottleneck isn't a Photon-eligible operator — tiny files, skew, or a UDF/scan of the whole lake.

Why: Check Photon usage % in the SQL profile. If it's high and you're still slow, it's I/O or shuffle skew. If it's ~0%, find the fallback expression. Don't buy bigger Photon SKUs first.

Senior depth: Photon accelerates a good plan. It does not fix a bad model, tiny files, or collect().

Common mistake: Upgrading DBU SKU without a profile.

Follow-up they will ask: What's the interview closer?

Lesson · Simulation

beginner

Where do you turn Photon on, and is there anywhere it is already on without you choosing?

Photon · tap to open the answer

Say this first: It is a compute choice: a runtime option when you create an all-purpose or job cluster, and part of the product on SQL warehouses and serverless compute.

Remember these

  • A runtime option on all-purpose and job clusters, with its own DBU rate
  • SQL warehouses and serverless compute run it as part of the product
  • No code, SQL or table changes are involved
  • Enabled on the cluster is not the same as used by the query

Why: Photon is not a library, a SQL dialect or a table format — it is an execution engine selected by the compute you run on. On all-purpose and job clusters you choose a Photon-enabled runtime at creation time, which is also where the higher DBU rate attaches. SQL warehouses run it as part of the product, so BI queries are already on it whether anyone decided that or not, and serverless compute is likewise Photon. Nothing about your SQL or DataFrame code changes, which is exactly why teams lose track of whether it is on. The place to verify is the query profile, not the cluster configuration page, because being on a Photon runtime does not mean a given query actually ran in Photon — unsupported operators fall back to the JVM on the same cluster.

Common mistake: Pointing at the cluster configuration as proof the workload is accelerated, without opening a single query profile.

Follow-up they will ask: Where in the profile would you confirm a specific query used it?

Lesson · Simulation

intermediate

You are told Photon is on. In the query profile, how do you prove how much of the query actually ran in Photon and find the exact fallback?

Photon · tap to open the answer

Say this first: Photon operators appear as Photon-prefixed nodes in the profile and plan; the fallback is the plain Spark node, and what matters is how much time is attributed to it.

Remember these

  • Photon-prefixed node names in the profile mark native execution
  • explain on a Photon cluster prints why parts were unsupported
  • Judge a fallback by the time attributed to it, not its presence
  • High coverage plus slow query means I/O, spill or skew, not Photon

Why: In the query profile, nodes executed natively carry Photon-prefixed names such as PhotonScan and PhotonGroupingAgg, and the profile attributes time and rows to each node, so you can see both what ran natively and what it cost. A fallback shows up as an ordinary Spark operator in the middle of the plan, usually with a transition around it. Running explain on a Photon-enabled cluster is the complementary check, because it prints the reasons Photon could not support parts of the query — that message names the specific expression or operator, which is what you need to fix it. The discipline is to look at time, not presence: a non-Photon node that holds 2% of the query's time is noise, and one that holds 80% is the whole investigation. If the profile shows high Photon coverage and the query is still slow, stop blaming the engine and go look at bytes scanned, file counts, spill and skew.

spark.sql(query).explain('formatted')
# Photon explain reports which operators it could not support, and why

Common mistake: Spotting one non-Photon node and declaring the query un-accelerated, when that node holds a sliver of the runtime.

Follow-up they will ask: Coverage is high, the query is still slow. What do you look at next?

Lesson · Simulation

intermediate

Photon bills a higher DBU rate. How do you decide whether it pays for a specific workload?

Photon · tap to open the answer

Say this first: Run the workload both ways and compare DBUs consumed, not wall clock — Photon pays when runtime falls by more than the rate multiplier.

Remember these

  • A speedup below the rate multiplier is a net cost increase
  • Compare DBUs from system.billing.usage, not just wall clock
  • Scan and aggregation heavy SQL clears the bar; UDF-heavy Python rarely does
  • Decide per workload; it is a per-cluster choice

Why: Because Photon raises the DBU rate per hour, a speedup smaller than the multiplier is a cost increase dressed as a performance win. So the test is arithmetic: run the same workload on the same data with and without Photon, and compare total DBUs from system.billing.usage alongside wall clock. Scan- and aggregation-heavy SQL over large Delta tables usually clears the bar comfortably, because that is exactly the operator set Photon implements natively. Workloads dominated by Python UDFs, RDD code or arbitrary notebook Python usually do not clear it, because there is little for Photon to accelerate and you pay the premium on all of it. Small queries are their own case: when startup and listing dominate, the engine barely matters. Decide per workload rather than per account, since you choose it per cluster and warehouses are on it regardless — and if latency rather than cost is the goal, say so explicitly, because then a modest speedup at a higher rate can still be the right call.

SELECT sku_name, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date > current_date() - INTERVAL 30 DAYS
GROUP BY sku_name ORDER BY dbus DESC

Common mistake: Comparing only wall-clock time, which makes a 20% faster query on a much higher rate look like a success.

Follow-up they will ask: The workload is 30% faster on Photon. Is that enough?

Lesson · Simulation

advanced

Which parts of a query does Photon actually accelerate, and name things that keep it out.

Photon · tap to open the answer

Say this first: Columnar scans of Parquet and Delta, projections, filters, hash aggregates, hash joins, sorts and writes — Python and pandas UDFs, RDD code, and some expressions and types fall back.

Remember these

  • Native: columnar scans, filters, projections, hash aggregates, hash joins, writes
  • Not native: Python and pandas UDFs, RDD code, some expressions and types
  • A SQL UDF is inlined, so its expressions can stay in Photon
  • Non-SQL Python work gets nothing from Photon at any price

Why: Photon reimplements physical operators as vectorized native code working on columnar batches, which is why its strengths are the operators where per-row JVM overhead dominated: reading columnar files, evaluating filters and projections, hash aggregation and hash joins. Outside that set, execution stays on the JVM path. Python and pandas UDFs are the clearest exclusion, since their semantics require a Python worker; RDD code is not part of the SQL engine at all; and individual expressions or data types that are not implemented natively will fall back even inside an otherwise supported operator. A SQL UDF behaves differently because it is inlined into the plan, so the expressions it contains can stay native — which makes rewriting a hot Python UDF as SQL a real engineering lever, not a cosmetic one. Knowing this boundary is mostly about not over-promising: a pipeline whose runtime is arbitrary Python will see almost nothing, and no DBU SKU changes that.

Senior depth: I frame Photon as plan coverage: it accelerates the parts of a query it implements, and the question is always what fraction of the runtime those parts hold. That makes the triage mechanical — sample query profiles, measure time in Photon nodes against total, and rank pipelines by the non-Photon time you could recover. It also sets expectations honestly with whoever is paying: Photon makes a good plan over Delta cheaper, and it does nothing for a data model problem, a tiny-file problem, or a pipeline of Python UDFs.

Common mistake: Promising a blanket multiplier for every workload, then explaining a flat result on a UDF-heavy pipeline after the invoice arrives.

Follow-up they will ask: How would you triage 30 pipelines to find the ones Photon will actually help?

Lesson · Simulation

advanced

There is one Python UDF halfway through an otherwise Photon plan. Does the whole query fall back, and what does the transition cost?

Photon · tap to open the answer

Say this first: No — Photon is per-operator, so the scan and aggregate around it can stay native, but each boundary converts columnar batches to rows and back.

Remember these

  • Photon eligibility is per operator, not per query
  • Each boundary converts columnar batches to rows and back
  • Python UDFs add worker serialization on top of the transition
  • Many scattered transitions can be slower than no Photon at all

Why: Photon decides operator by operator, so a single unsupported node does not disqualify the query; the scan beneath the UDF and the aggregate above it can both run natively. The cost is the boundary. Photon works on columnar batches and the JVM path works on rows, so crossing between them requires a conversion in each direction, and a Python UDF adds serialization to and from a Python worker on top of that. One such transition in a large scan-heavy query is usually worth paying because the scan is where the time was. Several transitions scattered through the plan can make the query slower than running it entirely on the JVM, which is the counter-intuitive result worth being able to explain. The remedies are ordered by effect: replace the UDF with built-in expressions so there is no boundary, push it to the very edge of the plan so only a small result set crosses, or precompute the column in silver so the interactive query never sees it.

-- replace the UDF with built-ins so no boundary exists
SELECT id, upper(trim(country)) AS country_norm
FROM ops.silver.orders

Senior depth: Transitions are the part people miss: the mental model is not Photon or JVM, it is a plan with boundaries that each cost a conversion. So when a UDF is unavoidable, I care about its position and its cardinality — applied to a small aggregated result at the top of the plan it is nearly free, applied per row on a billion-row scan it dominates and drags the conversion cost along with it. Precomputing the derived column once in silver is usually the cheapest answer, because it moves the cost out of every interactive query and into one batch job.

Common mistake: Assuming one UDF makes the whole query JVM-bound, so nobody bothers to look at where the time is actually spent.

Follow-up they will ask: The UDF cannot be rewritten in SQL. Where in the plan do you put it?

Lesson · Simulation

architect

A platform lead wants Photon enabled on every cluster in the account. What is your pushback, and what would you do instead?

Photon · tap to open the answer

Say this first: A blanket mandate raises the DBU rate on the whole fleet, including the clusters doing Python and ML work where Photon has almost nothing to run.

Remember these

  • The premium applies to uptime, not to accelerated plan fractions
  • Python, ML and RDD fleets pay the rate and gain almost nothing
  • Encode the decision in cluster policies per workload type
  • Low coverage plus high premium identifies a rewrite backlog

Why: Photon's rate premium applies to cluster uptime, not to the portion of the plan it accelerated, so enabling it on a cluster whose time goes to arbitrary Python, model training or RDD code is a straight cost increase with no corresponding speedup. The honest counter-proposal is segmentation: enable it where the work is SQL and DataFrame operations over Delta, which is most ETL and all BI, and leave the Python and ML fleet alone. Encode that in cluster policies by workload type so the default is right without anyone having to remember, rather than issuing an account-wide setting. Then measure instead of asserting: sample query profiles for the share of time in Photon nodes per workload, and compare DBUs per workload before and after from the billing usage tables. The pipelines that show high premium and low coverage are not an argument against Photon — they are a rewrite backlog, because UDF-heavy code is where the recoverable time is largest.

Senior depth: I would reframe it from a toggle to a portfolio decision with two measurements per workload: share of runtime in Photon nodes, and DBUs consumed. SQL and ETL over Delta gets Photon by policy default. Python, ML and RDD workloads get the standard runtime until something changes. And the UDF-heavy pipelines in the middle become a ranked backlog, because rewriting a hot Python UDF as built-in expressions pays twice — the operator becomes native and the plan loses its conversion boundaries.

Common mistake: Arguing the mandate on principle rather than bringing per-workload coverage and DBU numbers, which is the only thing that moves the decision.

Follow-up they will ask: What evidence would you collect before the next platform review?

Lesson · Simulation