Apache Kafka for Data & AI Engineers in Python

Setup Your Kafka Environment

Before we touch a single line of Kafka code, we need a place to actually run Kafka. This lecture is about getting that environment up and running on your own machine.

We're going to run a real Kafka cluster on your own laptop, using Docker. Not one broker — three. That's how Kafka is actually used in the real world, and I want you comfortable with that from day one.

By the end of this lecture, you'll have a 3-broker Kafka cluster running on your machine, and Python ready for the hands-on chapters ahead. Look at the picture below — that's exactly what we're building.

What you're about to buildWhat you're about to build

We'll do a quick sanity check at the end to confirm everything actually works, and then clean up and switch it off. The full hands-on — creating topics, producing and consuming messages — is what the next lecture is all about. Today is purely about getting the environment built and proving it actually runs.

What You Need Before Starting

  • A laptop or desktop with at least 8 GB RAM (16 GB is better)
  • Windows or Mac — steps for both are given below
  • A stable internet connection, just for a few one-time downloads
  • About 30 to 40 minutes of uninterrupted time

That's it. You don't need any cloud account for this. Everything runs locally, on your own machine.


Setting Up WSL2 (Windows Users Only)

If you're on a Mac, skip this section entirely and move on to installing Docker.

Docker on Windows needs something called WSL2 to run properly. It's a small piece of Windows that lets Docker work smoothly underneath the hood. You won't need to open or work inside it directly at any point in this course — Docker uses it quietly in the background, and every command you type from here on runs from your normal PowerShell.

  1. Open PowerShell as Administrator. Right-click the Start button and choose "Terminal (Admin)", or search "PowerShell", right-click it, and choose "Run as administrator".
  2. Run this command:
powershell
wsl --install

This one command turns on the Windows features WSL needs, installs WSL2 itself, and installs Ubuntu — a friendly, beginner-common flavor of Linux — as your default Linux system.

  1. Restart your computer when it asks you to. This step isn't optional.
  2. After restarting, check whether Ubuntu installed and opened on its own. If not, open PowerShell — regular this time, not admin — and run:
powershell
wsl -l -v
  • If you see Ubuntu in the list, good — it installed correctly. Search "Ubuntu" in the Start menu and open it yourself.
  • If you see a message like "Windows Subsystem for Linux has no installed distributions," install it directly:
powershell
wsl --install -d Ubuntu
  1. The first time Ubuntu opens as a real terminal window, it'll ask you to create a Linux username and a password. Keep it simple. When you type the password, nothing appears on screen — no dots, no stars. That's normal, not a glitch.
  2. Confirm you landed in the right place by running:
bash
pwd

You should see /home/yourname. You won't be doing your Kafka work inside this Linux home folder — Docker Desktop handles the connection to Windows on its own — but this confirms WSL is set up correctly.

Double-Check WSL2 Is Really Working

Open PowerShell again — regular, not admin — and run:

powershell
wsl --status

Look for Default Version: 2. If you instead see version 1, fix it with:

powershell
wsl --set-default-version 2

Installing Docker Desktop

Docker lets us run Kafka in isolated containers, without installing Kafka directly onto your machine. This keeps things clean, and it also matches how Kafka is actually deployed in real companies.

For Windows Users

  1. Go to https://www.docker.com/products/docker-desktop
  2. Download Docker Desktop for Windows
  3. Run the installer, keeping all default options selected — this includes the WSL2 backend we just enabled
  4. Restart your machine if asked
  5. Open Docker Desktop from your Start menu and wait until it says "Docker is running"

For Mac Users

  1. Go to https://www.docker.com/products/docker-desktop
  2. Download Docker Desktop for Mac — choose the correct version for your chip; the download page detects Apple Silicon vs. Intel automatically
  3. Open the downloaded file and drag Docker into your Applications folder
  4. Open Docker from your Applications folder and wait until it says "Docker is running"
  5. No WSL setup is needed on Mac — this step alone is enough

Confirm Docker Installed Correctly

This part is identical on both Windows and Mac. Open your terminal — PowerShell on Windows, Terminal on Mac — and type:

bash
docker --version

You should see something like Docker version 24.x.x, or higher.


Installing Python

This course uses Python, with the confluent-kafka library, starting a bit further into our hands-on chapters. Let's set it up now so it's ready when you need it.

For Windows Users

  1. Go to https://www.python.org/downloads/ and download the latest Python 3.x installer.
  2. Run the installer. On the very first screen, check the box "Add python.exe to PATH" before clicking Install. This box is easy to miss, and skipping it causes real problems later.
  3. Once installation finishes, close and reopen PowerShell — a fresh terminal is required to pick up the change.
  4. Confirm it worked:
powershell
python --version

You should see something like Python 3.12.x.

If you ever see a message like "Python was not found; run without arguments to install from the Microsoft Store" after this, it means Windows' own Store shortcut is intercepting the command instead of your real install. Go to Settings → Apps → Advanced app settings → App execution aliases, and turn off the toggles for python.exe and python3.exe.

For Mac Users

Mac comes with a system Python, but it's outdated and not meant for development use. Install a proper version instead:

  1. Go to https://www.python.org/downloads/ and download the latest Python 3.x installer for macOS.
  2. Run the installer, following the default prompts.
  3. Confirm it worked, in a new Terminal window:
bash
python3 --version

Installing the Kafka Python Library

This part is the same idea on both operating systems:

bash
python -m pip install confluent-kafka

On Mac, use python3 -m pip install confluent-kafka if plain python isn't recognized.


Creating Your Project Folder

Let's keep this course organized from day one.

  1. Create a new folder on your machine called kafka-course
  2. Inside it, create another folder called setup

We'll keep adding to this same setup folder as the course goes on — later chapters will ask you to drop new files in here, alongside what you set up today.


Downloading the Cluster Setup File

To start our 3-broker cluster, we need one configuration file called docker-compose.yml. This file tells Docker exactly how to set up our three brokers, using the official Apache Kafka Docker image — the same image published directly by the Apache Kafka project itself.

  1. Download docker-compose.yml from this lecture's resources on the learning platform
  2. Move the downloaded file into your setup folder

You don't need to open or understand this file right now — we'll go through it piece by piece in an upcoming lecture. For today, it's simply the file that brings your cluster to life.


Starting Your Kafka Cluster

Open your terminal, move into the setup folder, and run:

bash
docker compose up -d

The -d means "run in the background," so your terminal stays free.

The first time you run this, Docker will download the Kafka images. This can take a few minutes depending on your internet speed. That's completely normal — just let it finish.


Confirming Everything Actually Works

This is the sanity check for today — a quick way to prove your cluster is genuinely alive, not just that a command finished without error. We'll do the real, full hands-on with topics and messages in the next lecture.

First, confirm all three containers are up:

bash
docker ps

You should see three containers listed — kafka-1, kafka-2, and kafka-3 — each with a status of Up.

Now let's prove Kafka itself is actually responding, not just the containers. Open a terminal inside the kafka-1 container:

bash
docker exec -it kafka-1 bash

Your terminal prompt should change, showing you're now inside the container. Kafka's command-line tools live inside this container at /opt/kafka/bin/, but that folder isn't on the container's PATH by default. Fix that once, right after entering:

bash
export PATH=$PATH:/opt/kafka/bin

Now ask the broker to list its topics:

bash
kafka-topics.sh --list --bootstrap-server kafka-1:29092

Notice we're using kafka-1:29092 as the address, not localhost:9092. That's because we're inside the kafka-1 container right now, and localhost inside a container only means "this container." To reach the whole cluster, we use the network name Docker gave each broker — kafka-1, kafka-2, kafka-3 — on port 29092. You'll use this same style of address any time you're working with Kafka's tools from inside a broker container.

You'll see a short list of Kafka's own internal system topics — things like __consumer_offsets and __cluster_metadata, which exist automatically in every Kafka cluster. You haven't created anything of your own yet, and that's exactly the point: an empty-of-your-own-topics response here means the broker received your request and answered it correctly. Your cluster is genuinely alive.

Type exit to leave the container.


Cleaning Up

Since today's check didn't create anything of our own — no topics, no messages — there's nothing to remove. Cleanup today is simple: just switch the cluster off.

bash
docker compose stop

This switches your containers off and frees up your machine's resources, but keeps everything about your setup exactly as it is, ready to come straight back up next time.

docker compose stop is different from docker compose down. stop just pauses your containers. down removes them completely. Unless a future lecture specifically tells you to use down, always use stop.

The Routine You'll Use From Now On

bash
docker compose start # begin a session # ... do your hands-on work ... docker compose stop # end a session

Notice this is start, not up -d. You use up -d exactly once — the first time you create your containers, which you just did. From now on, start simply switches your existing containers back on, without recreating them or touching any data inside them.


Quick Recap

  • We're running a real 3-broker Kafka cluster locally, using Docker — the same way Kafka runs in production, just on your own laptop.
  • Docker Desktop, Python, and the confluent-kafka library are now installed and ready for the hands-on chapters ahead.
  • Your cluster lives in a setup folder, started from a single docker-compose.yml file you'll keep reusing.
  • docker compose up -d creates your containers — you only ever need this once. After that, it's start and stop.
  • We confirmed the cluster is genuinely alive by listing topics from inside kafka-1 — not just that containers are running, but that Kafka itself is responding.
  • This setup is what you'll keep using through the next several chapters. The next lecture is where we actually start using it — creating topics, and sending and reading real messages.