Choosing Azure Databricks compute
Compute is the biggest part of an Azure Databricks bill, so the exam often asks which type to use and which settings reduce cost without extra administration.
Compute types
| Type | Use it for | Notes |
|---|---|---|
| Serverless compute | Notebooks, jobs, pipelines, SQL | Databricks manages it, starts in seconds, no tuning |
| Job compute | Scheduled or automated runs | Created per run, ends when the run ends |
| All-purpose compute | Interactive development and shared exploration | Stays up until terminated, higher price |
| SQL warehouse | SQL queries, dashboards, BI tools | Pro, Classic and Serverless types |
When a question says “minimize operational cost and administrative effort”, the answer is usually serverless compute. When it describes a scheduled workload and serverless is not offered, the answer is a job cluster, not an all-purpose cluster.
Cost settings
- Autoscaling sets a minimum and maximum number of workers. The cluster adds workers under load and removes them when idle.
- Auto termination shuts down an idle all-purpose cluster after a set number of minutes.
- Fixed workers suits only steady workloads that never spike. If a job is predictable and the node type already fits, disabling autoscaling and trimming workers can cut cost.
- Photon is a faster vectorized engine. It speeds up SQL and DataFrame work, but it is not a way to scale down.
- Spot instances lower the price of worker nodes for fault-tolerant jobs.
Access modes and runtimes
Use standard or dedicated access mode so that Unity Catalog is enforced. Choose Databricks Runtime ML when you need machine learning libraries such as scikit-learn and TensorFlow preinstalled. A single-node cluster has a driver and no workers, which is fine for small data.
Libraries
Install libraries in a Unity Catalog friendly way: on the compute from Unity Catalog volumes or workspace files, using the compute Libraries tab or a policy. Running pip by hand in a notebook only affects that session.
Compute policies
A compute policy limits node types, maximum workers and required auto termination, and can set defaults. Assign policies to users or groups so they can create compute inside guardrails.
Exam traps
- All-purpose compute is for interactive work. Do not pick it for scheduled production runs.
- Autoscaling does not stop an idle cluster. You also need auto termination.
- Photon improves speed, not elasticity.
- A SQL warehouse is not for running Python notebooks or ML runtimes.
- Compute policies control what users may create. They do not scale clusters.
Practice questions
Choose an answer, then review the explanation to check your understanding.
1. A nightly notebook runs for 40 minutes and nobody uses the compute afterwards. Which compute is the cheapest fit?
Show answer
Answer: B.A job cluster starts for the run and is terminated when it ends, and job compute is billed at a lower rate than all-purpose compute.
2. A cluster must grow when load rises and shrink when it falls, at the lowest cost. Which two settings do you use?
Show answer
Answer: B.Autoscaling adjusts workers to the load, and auto termination stops the cluster after an idle period.
3. Analysts run BI dashboards with SQL only. Which compute should they use?
Show answer
Answer: C.SQL warehouses are built for SQL and BI workloads.
4. You want admins to stop users from creating oversized clusters. What do you use?
Show answer
Answer: A.Compute policies restrict and preset cluster settings such as node types, size limits and auto termination.