Apache Kafka For Absolute Beginners in Java

About the course

Welcome. This course teaches you Apache Kafka by letting you run it, watch it, and break it, using Java.

Most Kafka courses explain the ideas and stop there. Here, every idea is tied to something you can see on your own machine. You will run a real three-broker Kafka cluster with Docker, write Java programs that send and receive events, and then test what happens when things go wrong. You will stop a broker in the middle of a write, look at the files Kafka keeps on disk, rewind a consumer group, and send a bad message through a dead-letter topic.

By the end, you will know how Kafka works inside, and you will be able to build Java producers and consumers that you can trust.

Who this course is for

  • Data engineers who are moving from batch pipelines to real-time event streams
  • AI and ML engineers whose models and applications depend on fresh data
  • Java developers who have to build on, or take over, a Kafka pipeline
  • Engineers who already use Kafka and want to understand why it behaves the way it does, not just which method to call

You do not need any Kafka experience. Every idea is introduced through a real problem and built up one step at a time.

What you will be able to do

After this course, you will be able to:

  • Explain the difference between batch and stream processing, and when each one is the right choice
  • Describe how Kafka is built: topics, partitions, offsets, brokers, KRaft, replication, leaders, followers and the in-sync replica set (ISR)
  • Run and inspect a three-broker cluster and use the Kafka command-line tools
  • Configure retention and compaction, and see what each one does on disk
  • Write Java producers with keys, callbacks, custom partitioning and idempotent delivery, and tune them for throughput or durability
  • Write Java consumers with consumer groups, cooperative rebalancing and safe offset handling
  • Send bad messages to a dead-letter topic so they do not block the rest of your data
  • Reprocess past data by resetting a consumer group, or by building a replay tool that starts from a point in time

How this course is laid out

The course has five sections. The first four teach and build. The fifth is a capstone project where you use everything.

Section 1: Foundations

This section answers two questions: why does Kafka exist, and how do you get one running?

  • Batch and stream processing, and why the value of data drops the longer it waits
  • What Kafka is: a hub that separates the systems producing data from the systems using it, and a distributed commit log underneath
  • The three core roles: producer, broker and consumer, and how partitions and consumer groups let Kafka scale
  • A short tour of the wider Kafka ecosystem, so you know where the core pieces fit
  • Setting up your lab: Docker and a three-broker cluster on your own computer
  • The Kafka command-line tools, which you use to create topics, send messages and read them back

By the end of this section, you have a working cluster and you are comfortable using it.

Section 2: Inside Kafka

This section opens the box. You learn what Kafka does with your data and what it does when something fails.

  • Topics, partitions and offsets, and how one topic is spread across several brokers
  • KRaft, the controller and how the cluster agrees on its own state
  • Replication: leaders, followers and the in-sync replica set (ISR)
  • Log segments, the files Kafka writes to disk, and how to inspect them
  • Retention, which decides how long data is kept, and compaction, which keeps the latest value for each key
  • Two failure labs: you stop a follower broker and then a leader broker, and watch the cluster recover

By the end of this section, you can explain why Kafka keeps working when a broker does not.

Section 3: The Producer API

This section is about sending data into Kafka correctly. You work in Java with Kafka's official kafka-clients library — the same client the broker itself is built on.

  • Your first producer: serializers, delivery callbacks and flush()
  • Keys, and how a key decides which partition a message goes to
  • Writing your own partitioning logic
  • Delivery guarantees: retries, acks, min.insync.replicas and idempotent producers, which prevent duplicates
  • Tuning for throughput or durability with linger.ms, batch.size and compression
  • Handling a full producer buffer without losing messages

By the end of this section, you can write a producer and explain exactly what guarantee it gives.

Section 4: The Consumer API

This section is about reading data from Kafka safely and at scale.

  • The poll loop, group.id, auto.offset.reset and heartbeats
  • Consumer groups and how Kafka shares partitions between consumers
  • Rebalancing, and the difference between eager and cooperative-sticky assignment
  • Offsets and commits: when to commit, and how to avoid losing or repeating messages
  • Poison messages and how to move them to a dead-letter topic
  • Reprocessing the past: resetting a group's offsets, and building a replay tool that seeks to a point in time

By the end of this section, you can build a consumer that scales, survives restarts and handles bad data.

Section 5: Capstone project

This section brings everything together in one project. You are a new data engineer at ShopStream, an online shop, and the festival sale starts on Friday. The sales team needs live numbers, and the data is messy.

You build a real-time order pipeline:

  • Two producers that send order and fulfilment events, with your own partitioning rule, safe delivery and tuned batching
  • An order loader that checks every event, saves good ones in a database, and sends bad ones to a dead-letter topic
  • A sales aggregator that counts orders and revenue for each zone, every minute
  • A replay tool that rebuilds the numbers for any time window straight from Kafka

You also test your pipeline against failures. You stop a loader, crash a loader and stop a broker, and you check that no data is lost or counted twice.

The project comes in three parts: a document that explains the scope and requirements, a document that explains the design, and a step-by-step solution. You also get a starter kit with the data and a checker script, and the full solution code as a download. Plan for about 3 to 4 hours.

How to get the most out of this course

  • Type and run everything. Most lessons end in a lab on your own cluster. The understanding comes from doing the lab, not from watching it.
  • Do not skip the failure labs. The lessons where you stop brokers or send bad data teach you the most about how Kafka behaves in production.
  • Follow the order. Later labs reuse topics and consumer groups that you create earlier.
  • Start and stop your cluster the same way each time. Use docker compose start to begin and docker compose stop to finish.
  • Try the project before you open its solution. You learn more from getting stuck for a few minutes than from reading the answer straight away.
  • Read errors and logs first. When a lab does not work, the error message and the container logs usually tell you why.

A note on the lab environment

The cluster you build is a local learning setup. It is designed to be easy to run, inspect and break. It is not a production configuration, so do not use it as one.

Quick recap

  • Data loses value the longer it waits. Kafka exists so you can act on events while they still matter.
  • You learn Kafka hands-on, in Java, on a three-broker cluster that you run yourself.
  • The first four sections cover foundations, Kafka internals, the Producer API and the Consumer API.
  • The fifth section is a capstone project, with a starter kit and a full solution, where you build a real-time order pipeline.
  • Next, check the prerequisites so you know what to have ready before you start.