Databricks Compute Cluster

In this lecture, let's understand Databricks Compute — the cluster or computation capacity Databricks provides for running your code, pipelines, jobs, and data engineering workloads.

Two Types of Compute

Databricks offers two broad categories of compute:

1. Serverless compute
2. Classic compute

This lines up with what we already know about workspaces: a serverless workspace only offers serverless compute, while a classic workspace offers both serverless and classic compute.

Serverless Compute

Serverless compute is on-demand capacity — you don't need to know or care where the underlying server is. You simply run your code, pipeline, or workload, without managing cluster creation or termination yourself.

Key characteristics:

• No provisioning — you don't create a cluster; it's simply assigned to you.
• Fast startup — ready within a couple of seconds.
• Automatic scaling — scales up and down based on your workload, automatically.
• Lower cost — since there's no dedicated, always-on cluster sitting idle, serverless typically works out cheaper for most use cases.

Because of these advantages, serverless is recommended for most scenarios.

Classic Compute

With classic compute, you launch and manage the cluster: you define its size, scaling behavior, virtual machine types, and configuration. You launch it, and once you're done, you're responsible for terminating it.

Classic compute comes in two flavors:

Standard Cluster

• Can be accessed by multiple users — you grant access to whoever needs it.
• Best suited for collaborative projects, where multiple users or teams need to work on the same project using the same cluster.
• Also good for interactive data exploration — leave it running so people can query and explore data.
• Supports Scala workloads (serverless currently does not support Scala from notebooks).

Dedicated Cluster

• Scoped to a single user or a single group — this is the key difference from a standard cluster.
• Supports Scala, R, and RDD APIs.
• Supports GPU instances — important if you're doing heavy model training or other GPU-dependent work.

Quick rule of thumb: if you need Scala/R/RDD APIs, GPU support, or the work belongs to one person/team specifically, use a dedicated cluster. If multiple people/groups need shared access for collaboration or exploration, use a standard cluster. For most everyday work, serverless is still the recommended default.

Compute Menu: Serverless vs. Classic Workspace

If you look at the Compute menu in each workspace type, you'll notice a difference:

Serverless workspace — Compute menu offers:

• SQL Warehouse (mainly for data warehousing — not the focus of this course)
• Vector Search
• Apps
• Lakebase

Classic workspace — Compute menu offers all of the above, plus:

• All-purpose compute — where you actually create and manage dedicated/standard clusters.
• Job compute — clusters launched automatically for a specific job. You cannot create these manually; they're spun up when a job runs, and automatically terminated once the job finishes. We'll cover this more when we get to building pipelines and jobs.
• Pools — a pool of pre-provisioned virtual machines, used to speed up cluster launch time. Instead of waiting for new VMs to spin up, a cluster can grab already-running machines from the pool, launching almost instantly. This is tied to all-purpose clusters.
• Policies — rules that restrict what kind of clusters can be created (more on this below).

Understanding Cluster Policies

Policies control what settings are available to you when creating a cluster. Databricks provides a few predefined ones:

In a real organization, your workspace admin typically decides which policy you're allowed to use — you might only be permitted to create clusters under the "Personal Compute" policy, for instance. Admins can also create and assign custom policies. As an account admin (which you likely are, in your own learning environment), you'll have access to everything, including "Unrestricted" — but in a team setting, this is usually more locked down.

Creating an All-Purpose Cluster

Here's a walkthrough of the key settings you'll configure when creating an all-purpose (classic) cluster:

• Cluster name — whatever you'd like to call it.
• Policy — as discussed above (we'll use Unrestricted for full visibility into the options).
• Machine learning runtime — whether you want ML libraries pre-configured.
• Databricks Runtime version — different versions are available; the latest Long Term Support (LTS) version is typically recommended for stability (at the time of this recording, that's version 17.3, which includes Scala 2.13 and Spark 4.0.0).
• Photon acceleration — an optional performance boost, at extra cost. For learning purposes, you can usually leave this off to save cost.
• Worker type — the VM type for your cluster's nodes. Available types depend on your cloud provider (AWS, in our case), and vary by memory/CPU combination — for example, some are storage-optimized, some memory-optimized, some general-purpose, ranging anywhere from 8GB/2-core up to much larger configurations.
• Autoscaling — enable it and define a min/max worker count (e.g., min 2, max 8), or disable it and specify a fixed number of workers. You can also choose single-node — just one machine, no scaling at all.
• Auto-termination — the cluster terminates automatically after a period of inactivity (default is 120 minutes; you can lower this — e.g., 30 minutes — to save cost).
• Tags — optional labels for cost tracking/organization.
• Access mode (Advanced tab) — this determines who can use the cluster:
  • Dedicated — restricted to a single user or single group (supports all languages, including Scala/R, plus machine learning).
  • Standard — shared across multiple users/groups; supports Python, Scala, and SQL, but does not support machine learning on this access mode.
  • Auto — Databricks picks the most suitable mode automatically, based on your cluster type and policy.

Once configured, Databricks shows you an estimated cost in DBUs (Databricks Units) per hour — for example, a small single-node cluster might cost around 0.34 DBU/hour. Keep in mind: this is the Databricks cost only — your cloud provider (AWS) separately bills you for the actual virtual machine. Combined, a small cluster like this might run around $1/hour total.

An Important Trade-off: Startup Time

Here's something worth internalizing: classic clusters take time to launch — often 5–10 minutes, even for a small single-node cluster. Serverless, by contrast, is ready in seconds.

This matters a lot depending on your use case:

• If a cluster will run continuously for days or months, with a team using it regularly, the 5–10 minute startup time is a one-time cost — not a big deal.
• If you're trying to launch a cluster, run one job, and shut it down — that startup delay happens every single time, which adds real friction. In this scenario, serverless is clearly the better choice.

Using a Cluster From a Notebook

Once you have code in a notebook, you attach a compute source to run it — this could be:

• Serverless — just select it; no waiting.
• Serverless GPU — a serverless option with GPU support.
• A dedicated/standard cluster — must already be created and running (or in the process of starting).

Switching between compute sources on a notebook (say, from serverless to a dedicated cluster) clears the notebook's current state — variables and execution history reset, since you're now connected to a different compute environment entirely. The code itself doesn't change — only where it runs changes.

This decision — serverless vs. classic, and which specific cluster — is usually made by your organization, team, or platform admin, and appropriate access is granted to you accordingly, across your dev, test, and production environments.

A Practical Gotcha: Undersized Clusters

Worth knowing: if you create a cluster that's genuinely too small — say, a single-node cluster with just 2 CPU cores and 8GB memory — even a trivial piece of code (like creating a 3-row DataFrame) might take a very long time or fail to complete. That small a machine has to run the OS, the Spark framework, and your actual workload, all within a very tight resource budget — smaller than most personal laptops. If you're planning to use a dedicated/classic cluster for real work, size it appropriately — 4–8 CPU cores with proportional memory is a more reasonable starting point.

Cleanup: Don't Forget to Delete Your Cluster

This is important: classic clusters cost money for as long as they're running — in our example, roughly 0.34 DBU/hour plus the underlying VM cost. If you created a cluster just to follow along with this lecture, delete it before you move on, to avoid unnecessary charges. Cluster deletion itself is quick (a few seconds to initiate), though full resource cleanup in the background may take a couple of minutes.

Summary

The general guidance for this course: we'll prefer serverless compute for most of our work, and switch to classic compute (standard or dedicated) only when a specific lecture genuinely requires it — for example, when Scala support is needed. Whenever we do create a classic cluster for a demo, we'll make sure to clean it up afterward.

See you again. Keep learning, and keep growing!