Brokers, Clusters & Replication: Staying Coordinated and Durable

Here's where the payments story stands. It's split into 3 partitions, spread across 3 brokers, each one now handling its own share of the write load.

Partitions split across brokers, still one topic Partitions split across brokers, still one topic

Look at that picture again, but ask two different questions this time. First: these three brokers are each doing their own job independently — but are they actually behaving as one coordinated system, or are they just three separate machines that happen to store related data? And second, a quieter but more serious question: right now, each partition sits in exactly one place. What happens the day that one machine simply fails?

We're going to answer both, in order. Coordination first. Durability second.

Part 1: Are They Really One Cluster?

For these three brokers to genuinely function as a single cluster, some things need to be true at all times:

Every broker needs to agree on which partitions exist, and which broker owns each one.
If a broker goes down, something needs to notice, and something needs to decide what happens next.
If a new broker joins, something needs to fold it into the cluster properly.

This shared understanding — what the cluster looks like right now, at this exact moment — is called cluster metadata. The real question is: who's responsible for keeping it accurate, and making sure all three brokers agree on it?

Enter the Controller

Kafka's answer is to give one broker a special extra job. Among the brokers in a cluster, one of them also acts as the controller.

The controller is responsible for exactly the situations we just listed:

Tracking cluster membership — knowing which brokers are currently alive and part of the cluster
Detecting failures — noticing when a broker stops responding
Assigning partition leadership — deciding which broker becomes the leader for a partition, including reassigning it when a broker fails
Propagating metadata — making sure every broker in the cluster gets told about these decisions, so nobody is working off stale information

Without a controller, you'd just have brokers independently holding data, with no one actually keeping the cluster's shared picture of reality consistent.

But What Happens If the Controller Itself Fails?

Good question — and it's exactly the kind of question that exposes whether a design is actually solid. If only one broker could ever be the controller, and that specific broker crashed, the entire cluster's coordination would freeze.

This is where Raft comes in — and it's worth slowing down here, because this is a genuinely new idea if you're coming from typical data engineering tools. For this section, forget Kafka entirely. We're talking about a general-purpose algorithm, used across many different distributed systems, not something Kafka invented.

Raft is a consensus algorithm — a method that lets a group of machines agree on a shared sequence of decisions, reliably, even if some of those machines crash or temporarily disconnect. It works roughly like this:

The group elects one member as leader, for a period of time.
That leader proposes changes — new entries to add to a shared, ordered log.
Before a change is considered official, a majority of the group must acknowledge they've recorded it too.
If the leader disappears, the remaining members detect this and elect a new one, and the group carries on — without losing anything that was already agreed upon.

The whole point is this: as long as more than half the group is alive, the group can keep functioning correctly. No single machine, including the leader, is a point of failure.

(By the way — "Raft" isn't a random name. Its creators picked it as a play on words: a raft is literally something built out of logs, floating together. Consensus algorithms manage replicated logs of decisions, so the name is a quiet pun. You'll sometimes see it informally expanded as "Replication for Availability and Fault Tolerance," though that's a backronym, not its original meaning.)

That's Raft, in general. Now let's see exactly how Kafka puts it to work.

KRaft: How Kafka Actually Implements Raft

Kafka's specific implementation of this idea is called KRaft — short for Kafka Raft. Let's walk through exactly how it works, step by step.

The metadata log. Every decision about the cluster — a new topic being created, a partition's leader changing, a broker joining or leaving — gets written as an entry to a special, replicated log. This isn't an abstract idea; it's a real internal topic called __cluster_metadata, physically stored on disk, the same way any other topic's data is. If you ever look inside one of your broker containers' data directories, you'll find a folder named __cluster_metadata-0 sitting right there — that's it.

Normal operation. Among the brokers eligible to be controller (in our setup, all three), Raft elects one as the active controller — this is simply the Raft "leader" role, applied to this specific job. The active controller writes new entries to the metadata log and replicates them to the other controller-eligible brokers. Once a majority confirm they've stored an entry, it's considered committed — official, agreed-upon truth for the cluster.

Normal operation: metadata log replication Normal operation: metadata log replication

Now let's walk through what happens when the active controller fails — the exact scenario we just raised.

KRaft failover, step by step KRaft failover, step by step

Step 1: Broker 1, the active controller, crashes. The other brokers stop receiving its heartbeat.
Step 2: Broker 2 and Broker 3 — the remaining controller-eligible brokers — notice the silence and hold an election. Broker 2 requests votes; Broker 3 votes yes. That's a majority.
Step 3: Broker 2 becomes the new active controller. It picks up exactly from the last committed entry in the metadata log — nothing is lost, because that entry was already replicated to a majority before Broker 1 went down.

This is exactly what makes a Kafka cluster genuinely fault-tolerant at the metadata level, not just at the data level.

A Quick Side Note: How This Used to Work

Before KRaft existed, Kafka solved this same problem differently — by relying on a completely separate piece of software called ZooKeeper to store and coordinate cluster metadata.

The old way: a separate ZooKeeper ensemble The old way: a separate ZooKeeper ensemble

This worked, but it meant running and maintaining two distributed systems side by side, just to have one working Kafka cluster — more infrastructure, more monitoring, more things that could fail independently of Kafka itself. You'll still encounter ZooKeeper if you ever work with an older Kafka deployment, which is why it's worth knowing by name — but every modern cluster, including yours, uses KRaft instead.

Where We Stand

Splitting payments across three brokers meant those brokers needed a way to stay coordinated as one system. Kafka solves this with a controller, responsible for tracking membership, detecting failures, and assigning partition leadership. Because a single controller would itself be a point of failure, Kafka uses the Raft consensus algorithm — as KRaft — writing every cluster decision to a replicated metadata log, so the controller role can survive a failure by electing a new one, without losing the cluster's agreed-upon state.

We now know who's in charge of the cluster's shared metadata. That's coordination. Now let's turn to the second question we opened with — what happens to the actual data sitting in each partition, if the broker holding it goes down.

Part 2: One Copy, One Point of Failure

Right now, each partition exists in exactly one place. Partition 0 lives only on Broker 1. Partition 1 lives only on Broker 2. Partition 2 lives only on Broker 3.

Broker 2 is a machine. Machines fail. Disks die, containers crash, hardware gives out — it happens, eventually, to every machine, no matter how well it's maintained.

One copy, one point of failure One copy, one point of failure

Think about what that actually means for a payments company. If Broker 2's disk fails, every payment event that ever went into Partition 1 is gone. Not delayed. Not recoverable. Gone — including transactions your finance team needs for reconciliation, and records your fraud team may need for an investigation. For a system handling money, that's not an inconvenience. That's the kind of failure that ends up in a legal conversation.

Notice that this is a completely different problem from the one we just solved. KRaft protects the cluster's shared understanding of itself. It does nothing to protect the actual payment events sitting on one disk. We need more than one copy of each partition.

The Fix: Replication

The fix is conceptually simple: instead of storing each partition once, store it multiple times, on different brokers. If payments uses a replication factor of 3, that means every partition — Partition 0, 1, and 2 — exists as 3 identical copies, spread across the cluster, not just 3 partitions spread once each.

Replication factor 3 — every partition copied onto all 3 brokers Replication factor 3 — every partition copied onto all 3 brokers

Look closely at that picture. This is the part that surprises people the first time they see it: every broker ends up holding a copy of every partition — not just its "own" one. Broker 1 doesn't just hold Partition 0 anymore; it also holds a copy of Partition 1 and a copy of Partition 2. The same is true for Broker 2 and Broker 3. Now, if any single broker's disk fails, the other two brokers still have every partition's data, completely intact.

Leader & Follower: Not All Copies Are Equal

So now Partition 1 has 3 copies — one on each broker. Here's a question worth pausing on: if a producer wants to send a new payment event to Partition 1, which of those 3 copies should it actually write to?

If the answer were "any of them, pick one," you'd have a mess almost immediately — different brokers accepting different writes in different orders, with no way to agree on what actually happened first. So Kafka doesn't allow that. For each partition, exactly one of its copies is designated the leader. The rest are followers.

Every producer and consumer for that partition talks only to the leader. Producers send new messages to the leader. Consumers read from the leader. The followers' entire job is to continuously copy whatever the leader has, staying ready to take over instantly if the leader ever goes down.

Partition 1's leader and followers Partition 1's leader and followers

Notice this lines up exactly with the matrix from a moment ago: Broker 2 was shown as the leader for Partition 1 — and that's exactly what you see here. Broker 1 and Broker 3 hold the same data, but only as followers, quietly staying in sync in case they're ever needed.

This is why a leader exists: one single point of truth per partition keeps writes ordered and consistent. And it's why followers exist: not to serve traffic, but to be an instantly-ready backup, so a broker failure doesn't mean a scramble to rebuild lost data from nothing.

The ISR: Kafka's Honesty Mechanism

There's one more piece, and it's the one that actually matters most in production. "Follower" tells you a broker should have a copy. It doesn't tell you whether that copy is actually caught up right now.

A follower can fall behind — a slow disk, a network hiccup, a broker that just restarted and is still catching up. Kafka doesn't pretend everything is fine when that happens. It tracks exactly which replicas are genuinely keeping pace, in a list called the ISR — the In-Sync Replica set.

A follower falls behind A follower falls behind

Broker 3 still technically holds a copy of Partition 1. But it's fallen behind, so Kafka removes it from the ISR — the list honestly shrinks from [1, 2, 3] to [1, 2].

Why does this matter so much? Two concrete reasons:

Durability guarantees depend on it. A producer can ask Kafka to only confirm a write once it's been copied to every in-sync replica, not just accepted by the leader. That guarantee is only as good as the ISR being accurate.
Leader elections depend on it. If Broker 2 (today's leader) ever fails, Kafka must promote a new leader only from the ISR. Promoting a replica that had fallen behind would silently lose whatever data it missed. The ISR is what keeps that from happening.

Bringing Both Halves Together

Step back and look at the whole shape of this lecture. Coordinating three brokers as one cluster, and keeping one partition's data safe across three brokers, turned out to be two separate problems — solved by two separate mechanisms.

KRaft keeps the cluster's shared metadata consistent, so every broker agrees on what the cluster looks like, and that agreement survives a controller failing. Replication keeps the actual data safe, by storing every partition on multiple brokers, with one leader taking writes and reads, followers quietly staying in sync, and the ISR honestly tracking who's genuinely caught up.

That combination — a coordinated cluster, and durable, replicated data — is what makes Kafka survive real machine failures without losing anything or grinding to a halt. It's also the honest, mechanical answer to one of the most common Kafka interview questions: "What happens if a broker fails?"