Introduction to Stream Processing
Let's start with a story.
Imagine you work at a bank. Every night, at 2 AM, a job runs that looks at all of yesterday's transactions and checks for fraud. It's a good system — it catches a lot of bad transactions.
But here's the problem. If someone steals a card at 9 AM and starts making fraudulent purchases, your fraud check won't even look at that transaction until 2 AM the next day. By then, the money is gone. The fraud check did its job — it just did it too late to matter.
Look at the timeline below. Seventeen hours pass between the moment the fraud happens and the moment anyone even looks at it. That gap is the entire problem in one picture.
A stolen card: 17 hours in the life of one transaction
This gap — the time between an event happening and someone actually acting on it — is the difference between batch processing and stream processing. And it's the first idea this whole course is built on.
Two Ways to Process Data
There are really only two ways to process data, and once you see the difference, you'll start noticing it everywhere.
Batch processing: wait, then process
Batch processing means you collect data for a while, and then process all of it together, at some fixed point — every night, every hour, every few minutes.
Think of it like laundry. You don't wash one sock the moment it gets dirty. You throw dirty clothes in a basket all week, and then, on laundry day, you wash everything at once. That's batch thinking: let things pile up, then handle them together.
And honestly, this works fine for a lot of things. Monthly sales reports. Weekly inventory counts. Nobody needs those calculated the instant a sale happens — a delay of a day, or even a week, costs nothing.
Stream processing: handle it the moment it happens
Stream processing flips that completely. Each piece of data is handled the moment it arrives — not later, not in a batch, right now.
Go back to the fraud example. With streaming, the moment that stolen card is used, the system checks it instantly and can block the transaction before it even completes. No overnight wait. No batch. Just: event happens, event is handled.
The diagram below puts these two side by side. In the batch lane, events pile up along the timeline, and then one processing job fires at a fixed time and handles all of them together. In the streaming lane, every single event gets its own arrow straight down to being processed — instantly, one at a time, as it happens.
Batch vs streaming processing flow
Why Timing Matters: The Value of Data Decays
Here's the idea that ties all of this together, and it's worth sitting with for a moment.
Data has a shelf life. The moment something happens — a click, a transaction, a sensor reading — that piece of data is at its most valuable. The longer you wait to act on it, the less that value is worth. For some data, the decay is slow: a monthly sales number is still just as useful a week later. For other data, the decay is brutal: fraud data is worth almost nothing 17 hours later, because by then the money's already gone.
Look at the curve below. It starts high on the left — the instant the event happens, acting on it is worth the most it will ever be worth — and it drops fast. The two points marked on the curve are our same fraud example: catch it in seconds, and you block the charge before it clears. Catch it 17 hours later, and you're just documenting a loss.
The value of data decays over time
This curve is really the only question you need to ask when deciding between batch and streaming: how fast does the value of this data disappear? If the answer is "slowly," batch is fine, and often cheaper and simpler. If the answer is "in seconds," you need streaming.
So Why Does This Matter for You?
Here's the honest answer: most of the data world ran on batch for decades, and it worked fine. Streaming isn't "batch, but better" in every situation. It's the right tool for a specific kind of problem — when the value of data drops fast the longer you wait to act on it.
A few examples that matter directly for this course, and that we'll come back to again and again:
- Fraud detection — the value of catching it drops to zero the moment the money leaves the account.
- Recommendations — knowing what a customer clicked 2 seconds ago is far more useful than knowing what they clicked yesterday.
- Change Data Capture (CDC) — keeping a data warehouse in sync with a live application database, so analysts aren't working off yesterday's numbers.
- Keeping AI systems fresh — an AI system answering questions from a knowledge base that updates once a day will confidently give you outdated answers. Streaming keeps that knowledge current.
Look at these four side by side. Different industries, different problems, but the same underlying reason each one needs streaming: the moment the data goes stale, whatever you build on top of it — a decision, a recommendation, an AI-generated answer — goes stale with it.
Where streaming earns its keep — four use cases
Two Tools, Not One Winner
None of this means batch is obsolete. Plenty of real, important work is still done in batch, and will be for a long time — batch is simpler, cheaper, and easier to reason about, and most data doesn't lose its value in seconds.
What it means is that you now have two tools, and a big part of being a good data engineer — or a good AI engineer building on live data — is knowing which one a problem actually needs. Not every pipeline should be a streaming pipeline. The question is never "should I use streaming?" in general. It's always "how fast does this specific data lose its value?" Answer that, and the choice between batch and streaming makes itself.
Quick Recap
- Batch processing collects data and processes it together at fixed intervals. Simple, cheap, and fine for most reporting.
- Stream processing handles each event the instant it arrives. You need it when delay costs you something real.
- The deciding factor is always the same: how quickly does the value of this data decay?
- Streaming shows up everywhere fast-decaying value matters: fraud detection, recommendations, CDC, and keeping AI systems fresh.
- Batch isn't going away. This is about having two tools in your kit and knowing when to reach for which one.