Configuring Retention and Compaction

Back in an earlier lecture, we looked at the two ways Kafka cleans up old segments — time-based retention, and compaction — as diagrams. Today, you're going to configure both, for real, and watch each one actually remove data on its own, live on your cluster.

We'll do retention first, on the payments topic you already have. Then we'll build a fresh changelog-style topic, account-balances, and watch compaction work.

Demo 1: Time-Based Retention on `payments`

Step 1: Two Configs, Two Very Different Rules

Before touching anything, it's worth knowing this, because it explains why this demo involves some waiting:

retention.ms is a per-topic setting. You can change it any time with kafka-configs.sh, on a topic that already exists, and it takes effect immediately — no restart needed.
log.retention.check.interval.ms is a broker-level setting — how often the broker's background cleanup thread even bothers to check for expired segments. This one is not dynamically alterable — it's only read once, when the broker starts, from its configuration. Changing it means editing your cluster's setup and restarting the brokers.

We're going to leave that second one exactly as it is — the default, 5 minutes — rather than touch the cluster's configuration. That means this demo has a real waiting period built into it: once we shrink payments' retention window, we still have to wait for the broker's next scheduled check before anything actually gets deleted.

Step 2: Shrink `payments`' Retention Window

This part is dynamic, so no restart needed:


bash
kafka-configs.sh --bootstrap-server kafka-1:29092 \
  --entity-type topics --entity-name payments \
  --alter --add-config retention.ms=10000

This tells Kafka: once every message in a sealed segment is more than 10 seconds old, delete the whole file. The 10-second window itself takes effect immediately — what we're waiting on is the broker's next check cycle, not this.

Step 3: Produce Some Fresh Messages — Into Partition 1

payments already has most of its data sitting in Partition 1, from the leader-election lab earlier in this chapter, where every message was sent with the key c to deliberately land there. Let's keep using that same key here, so the new messages stack onto the same partition — that way, the "before" and "after" picture you're about to compare are both looking at the partition that actually has meaningful data to lose, not an empty one.


bash
( for i in $(seq 1 50); do echo "c:payment-event-$i"; sleep 0.1; done ) | \
kafka-console-producer.sh --topic payments --bootstrap-server kafka-1:29092 \
  --reader-property "parse.key=true" \
  --reader-property "key.separator=:"

Step 4: Check the Segments Right Now


bash
ls -la /var/lib/kafka/data/payments-1

Note how many .log files exist at this moment — this is your "before" picture.

Step 5: Wait for the Broker's Next Check, Then Look Again

This is the real-time part. With the check interval at its default, the broker won't even look for expired segments for up to 5 minutes. Pause here for about 5 minutes, then run the same command again:


bash
ls -la /var/lib/kafka/data/payments-1

Compare it to Step 4. The older sealed segments — the ones whose messages are now all more than 10 seconds old — should be gone. Only recent segments, and the current active one, remain. The active segment is never deleted this way, no matter how old the topic is, since it's still being written to.

This is exactly the mechanism from earlier in this chapter, happening in front of you: no editing, no partial deletion — a whole segment file, gone in one shot, the moment it fully aged out, on the broker's own schedule.

Step 6: Restore `payments`' Default Retention

This override doesn't clean itself up — it's stored as part of the topic's config, not just this session, so it'll silently keep deleting payments' data every 10 seconds for every lecture after this one unless you remove it now:


bash
kafka-configs.sh --bootstrap-server kafka-1:29092 \
  --entity-type topics --entity-name payments \
  --alter --delete-config retention.ms

This puts payments back on Kafka's normal 7-day default. Confirm the override is actually gone:


bash
kafka-configs.sh --bootstrap-server kafka-1:29092 \
  --entity-type topics --entity-name payments --describe

You shouldn't see retention.ms listed as a dynamic override anymore.

Demo 2: Compaction on `account-balances`

Earlier in this chapter, we imagined a second topic alongside payments — one called account-balances, representing current state instead of event history. Now let's actually build it, and watch Kafka's compaction quietly remove outdated duplicates on its own.

Step 1: Open a Terminal Inside Broker 1

(Skip this if you're continuing straight on from Demo 1 — you're already there.)


bash
docker exec -it kafka-1 bash
export PATH=$PATH:/opt/kafka/bin

Step 2: Create `account-balances` With Compaction Enabled

We're using one partition here, just to keep today's demonstration easy to follow — compaction works exactly the same way regardless of partition count. We're also deliberately using aggressive settings so compaction runs quickly enough to observe in this lab, instead of waiting on production-scale timers.


bash
kafka-topics.sh --create \
  --topic account-balances \
  --bootstrap-server kafka-1:29092 \
  --partitions 1 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config segment.ms=100 \
  --config min.cleanable.dirty.ratio=0.01 \
  --config delete.retention.ms=100

In production, you would typically only set cleanup.policy=compact and leave the timing settings at their defaults — we're only tightening them here to speed up today's demo.

Output / Note

Note: You might expect a segment.bytes setting here too, to force tiny segments by size. On current Kafka versions, segment.bytes has a hard minimum of 1,048,576 (1 MB) — smaller values are rejected outright. segment.ms=100 alone is enough to roll segments fast here, so we leave segment.bytes at its default. This is the same size-based limitation we're deliberately leaving out of this lecture — we're only configuring what's actually needed for each demo.

Step 3: Produce Balance Updates — With Repeating Keys

This is the important part. Unlike payments, every message here needs a key — the customer ID whose balance changed. Start a keyed producer:


bash
kafka-console-producer.sh --topic account-balances --bootstrap-server kafka-1:29092 \
  --reader-property "parse.key=true" \
  --reader-property "key.separator=:"

Type these lines one at a time, pressing Enter after each — notice user1 and user2 each appear more than once, simulating their balance changing over time:

user1:100
user2:50
user1:150
user3:75
user1:200
user2:80

Press Ctrl+C to exit the producer once you've sent all six.

Step 4: Give Kafka a Moment to Compact

Compaction runs on a background cycle, not instantly on every write. Wait about 30 to 60 seconds before continuing.

Step 5: Consume From the Beginning


bash
kafka-console-consumer.sh --topic account-balances --bootstrap-server kafka-1:29092 \
  --from-beginning \
  --formatter-property print.key=true \
  --formatter-property key.separator=" : "

If compaction has already run, you'd expect to see only three messages — one per key, each holding its latest value:

user1 : 200
user2 : 80
user3 : 75

In practice, you may well see four instead — user2's old value (50) sitting alongside its new one (80), with user1 and user3 each correctly collapsed. That's not a broken demo — it's a direct, honest consequence of a rule from earlier in this chapter: Kafka's cleaner never touches the currently active segment, only sealed ones. user2:80 was your very last message. There's a good chance it's still sitting in whatever segment is currently active — and a segment only seals when a new message arrives afterward and triggers that check. Since nothing else has been produced since, that segment simply has no reason to close yet, so its contents — including user2:80 — haven't been considered for compaction at all. user2:50, sitting alone in an already-sealed segment with no newer copy visible to the cleaner yet, is correctly left exactly where it is. This will resolve itself naturally the moment any new message is produced to this topic in the future, sealing that segment and making user2:80 eligible for the next compaction pass.

Step 6: See It on Disk — and Read What Compaction Actually Did

This is where compaction stops being a claim and becomes something you can verify with your own eyes. List the segments:


bash
ls -la /var/lib/kafka/data/account-balances-0

You'll likely see two segments: the first one, sealed, holding most of your messages — and a second, active one, holding just the last message or two (this is the same active-segment behavior Step 5 just described). Dump the first, sealed segment:


bash
kafka-dump-log.sh --files /var/lib/kafka/data/account-balances-0/00000000000000000000.log --print-data-log

Here's a real example of what that looks like:

Dumping /var/lib/kafka/data/account-balances-0/00000000000000000000.log
Log starting offset: 0
baseOffset: 1 lastOffset: 1 ... key: user2 payload: 50
baseOffset: 3 lastOffset: 3 ... key: user3 payload: 75
baseOffset: 4 lastOffset: 4 ... key: user1 payload: 200
Non-consecutive offsets in /var/lib/kafka/data/account-balances-0/00000000000000000000.log
  -1 is followed by 1
  1 is followed by 3

Remember the six messages you sent landed at offsets 0 through 5, in order: 0=user1:100, 1=user2:50, 2=user1:150, 3=user3:75, 4=user1:200, 5=user2:80. Offset 5 is in the second, active segment — but look closely at what's missing from this one. Offsets 0 and 2 — the two superseded user1 values, 100 and 150 — simply aren't there anymore. Only offset 4, user1's latest value at the time this segment was cleaned, survived. kafka-dump-log.sh even tells you this directly: the "Non-consecutive offsets" warning is the tool noticing the same gap you are. That gap is compaction's fingerprint.

Notice user2:50 (offset 1) and user3:75 (offset 3) are both still here, untouched — exactly as expected. user3 never had a duplicate, so there was nothing to remove. user2's newer value was in the still-open second segment when this cleaning pass ran, out of the cleaner's reach, so user2:50 correctly remained the only known value at the time.

Does this actually free up space, though — the file's still there, same name? Yes, genuinely. Kafka doesn't edit a segment in place — it reads the whole segment, builds a brand new file containing only the survivors, and atomically swaps it in under the same base-offset filename. Look at the position values in the dump: 0, 75, 150 — three records, sitting back-to-back with no gaps. If this file still physically contained all five of its original messages, you'd see five entries here, not three, and the position values would reflect that larger, unedited file. The smaller, renumbered positions are direct evidence the file was rewritten smaller — the two removed records aren't sitting somewhere marked "deleted" inside a still-large file; the bytes they used to occupy are simply gone, reclaimed when the replacement file was written. The filename staying constant is just Kafka's naming convention — segments are always named after the offset they start at, and that starting offset didn't change even though what's inside did.

Step 7: Clean Up


bash
docker compose stop

Checklist

Understood why retention.ms (per-topic, dynamic) and log.retention.check.interval.ms (broker-level, static) behave so differently
Shrank payments' retention.ms to 10 seconds
Produced fresh messages into Partition 1, where payments' data already lives
Waited out a real broker check cycle and watched old segments actually disappear from disk
Restored payments' retention.ms back to the default and confirmed the override is gone
Created account-balances with cleanup.policy=compact
Produced keyed messages with repeated keys
Consumed the topic and understood why the very last key's old value might still linger, thanks to the active-segment rule
Dumped a sealed segment and used the gap in offsets as direct proof of what compaction removed

That completes the hands-on side of this chapter's architecture. You've now built, broken, recovered, and cleaned up a real Kafka cluster — not just read about one.