Notebook Magic Commands

In this lecture, let's learn about notebook magic commands — what they are, why they exist, and the most commonly used ones for data engineering work.

Setup: Attaching a Dedicated Cluster

Go to your workspace explorer, create a folder called exercises (if you haven't already), and inside it, create a new notebook — let's call it 02-magic-commands.

Before running anything, attach a cluster. This time, we'll use a dedicated cluster instead of serverless. Here's why: dedicated clusters support Python, SQL, Scala, and R, while serverless clusters currently support only Python and SQL — no Scala. Since this lecture covers a Scala magic command, we need a dedicated cluster.

What Is a Magic Command?

A magic command is a special keyword that starts with a percent sign (%). It adds extra capabilities to the notebook environment — capabilities that go beyond what the language itself (Python, SQL, Scala) can do on its own.

There's a long list of magic commands in general, but as a data engineer, you really only need to know a handful. Let's go through them.

A quick way to see what's available: run %lsmagic in a cell — it lists all supported magic commands. You can also check documentation for a specific one using %fs? (short help) or %fs?? (more detailed help).

pythonCopy
%lsmagic

pythonCopy
%fs?

pythonCopy
%fs??

Language-Switching Magic Commands

The first, and most useful, category of magic commands lets you switch the language of a single cell — without changing the notebook's overall default language.

%python

Our notebook's default language is already Python, so %python isn't strictly required here — but it's useful when your notebook's default language is something else (like SQL or Scala) and you want to drop into Python for a specific cell.

Let's create some sample data and register it as a temporary view, so we can query it with other languages later:

pythonCopy
%python

employee_data = [
    ('Alice',  'Engineering', 95000),
    ('Bob',    'Marketing',   72000),
    ('Carol',  'Engineering', 105000),
    ('David',  'HR',          65000),
    ('Eva',    'Engineering', 98000),
]
columns = ['Name', 'Department', 'Salary']

emp_df = spark.createDataFrame(employee_data, columns)

# Register as a SQL temp view so %sql cells can query it
emp_df.createOrReplaceTempView('employees')

Run this, and it creates a temporary view named employees in the Databricks metadata catalog (Unity Catalog).

%sql

Now, even though our notebook's language is Python, we can still run SQL directly in a cell. Just add %sql at the top — this changes the cell's language to SQL, while the notebook's overall default language stays Python.

sqlCopy
%sql

SELECT
    Department,
    COUNT(*)        AS HeadCount,
    AVG(Salary)     AS AvgSalary,
    MAX(Salary)     AS MaxSalary
FROM employees
GROUP BY Department
ORDER BY AvgSalary DESC

Run it, and you get a proper query result — grouped, aggregated, and sorted — querying the temp view we created in Python moments earlier. This is the real power of magic commands: you can mix languages freely within the same notebook, using whichever is most convenient for a given task.

%scala

Similarly, %scala switches a cell's language to Scala:

scalaCopy
%scala

val greeting = "Hello from Scala!"
println(greeting)
println(s"Spark version: ${spark.version}")

Running this prints the greeting and the current Spark session version (e.g., Spark version: 4.0.0) — real Scala code, running inside a notebook whose default language is Python.

%md

%md switches a cell to Markdown, letting you write formatted documentation directly inside your notebook — headings, tables, bold text, and so on. For example, this very notebook opens with a markdown cell like this:

02 — Magic Commands in Databricks Notebooks

Purpose

Demonstrates all core Magic Commands used in Databricks data engineering.

Magic Commands Covered

Author: ScholarNest | Last Updated: 2026

Run this cell, and instead of raw text, you get a nicely rendered document — headings, a table, and bold text, just like you'd expect from Markdown anywhere else.

The takeaway: notebooks let you mix Python, SQL, Scala, and documentation, all in the same place — and the way you do that is through these language-switching magic commands: %python, %sql, %scala, and %md.

File System Magic Command: %fs

The next important one is %fs — the file system magic command. It gives you access to Databricks File System (DBFS) commands, directly from a notebook cell.

Listing Directories

pythonCopy
%fs
ls /databricks-datasets/Rdatasets/data-001/csv/ggplot2/

A couple of notes on paths: the Databricks file system qualifier is dbfs, but you don't need to type it explicitly — if you just use a path starting with /, Databricks automatically assumes you mean DBFS. Running %fs ls / at the root shows you the top-level directories in this environment. You can drill further down — for example, into /databricks-datasets/Rdatasets/data-001/csv/ggplot2/, which contains a set of sample CSV files, including the diamonds.csv file we've used in earlier lectures.

Reading File Contents

pythonCopy
%fs head /databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv

The head command works like the Linux head command — it reads and displays the first few lines of a file, without loading the whole thing.

Creating Directories

pythonCopy
%fs mkdirs /Workspace/Shared/Examples

This creates a new directory — in this case, an Examples folder inside your workspace's Shared area. If you check your workspace explorer afterward (refreshing if needed), you'll see the new folder there.

%fs supports several other commands too — for listing, creating, deleting, and copying files/directories within DBFS. They work much like familiar Linux shell commands, but operate specifically within the Databricks file system. You won't use these constantly, but they do come up regularly enough in real projects that it's worth knowing they exist.

Shell Magic Command: %sh

%sh lets you run actual Linux shell commands directly from a notebook cell:

pythonCopy
%sh ls /

Note: only non-interactive shell commands work here — anything requiring interactive input isn't supported from a notebook cell (you'd need an actual terminal for that).

Important distinction: when you run %sh ls /, you're looking at the driver node's local file system — not DBFS. This is a completely different filesystem from what %fs ls / shows you. Don't confuse the two: %fs operates on DBFS, while %sh operates on the driver node's own Linux shell.

Running Other Notebooks: %run

The last, and one of the most powerful, magic commands is %run. It lets you execute another notebook from within your current notebook — and anything defined in that other notebook (functions, variables) becomes available in your current notebook's context.

Let's see this in action. Create a second notebook called 03-common-utils, and define a simple utility function in it:

pythonCopy
def get_env_config():
    """Returns environment configuration as a dictionary."""
    return {
        'raw_path':    '/Shared/dbx-de-course/raw/',
        'silver_path': '/Shared/dbx-de-course/silver/',
        'gold_path':   '/Shared/dbx-de-course/gold/',
        'env':         'development'
    }

Back in 02-magic-commands, use %run to bring in that notebook — since both notebooks live in the same directory, we just reference it by relative path:

pythonCopy
%run ./03-common-utils

This cell won't show any visible output, because 03-common-utils doesn't produce output itself — it just defines a function. But once this cell runs, get_env_config() becomes available in our current notebook, even though we never defined it here ourselves:

pythonCopy
config = get_env_config()
print('Environment config:')
for key, value in config.items():
    print(f'  {key}: {value}')

Run this, and it works — printing each config key and value. The function came from 03-common-utils, made available here purely through %run.

This is genuinely powerful for real projects: it lets you write modular code — common utilities, shared configuration, reusable logic — in separate notebooks, and then pull them into a driver/master notebook that orchestrates the overall flow. This is a common pattern for building multi-notebook pipelines and applications in Databricks.

Summary

These are the core magic commands you'll rely on most often as a data engineer working in Databricks notebooks.

See you again. Keep learning, and keep growing!