Module 3: Error Handling, Retries, Resilience

What you'll learn in this module

By the end of this module, you'll be able to:

• Tell the difference between a failure worth retrying and one that never will succeed, no matter how many times you try it
• Explain why retrying blindly can make a bad situation worse, not better
• Add real backoff and jitter to a retry loop, and explain what each one is actually protecting against
• Build a classified exception hierarchy your retry logic can actually make decisions on, instead of catching "something went wrong"
• Give failed records somewhere to go — a dead-letter path — instead of crashing the whole run or silently dropping them
• Spot the single most common, most damaging retry mistake AI assistants make, and catch it before it ships

Let's begin.

---

Let's start with a question

Have you ever seen a system retry the same failing request, again and again, for no reason at all?

Not a network blip. Not a server having a bad moment. A request that was wrong from the start — a bad password, a malformed field, a currency that doesn't exist. No amount of retrying was ever going to fix that. But the code didn't know the difference. It just saw an error, and tried again. And again. And again.

Here's the uncomfortable part. This isn't a rare mistake. It's one of the most common bugs in real production systems — and it's not just wasteful. At real scale, it can turn a small, contained problem into an outage that spreads far past where it started.

This module is about knowing the difference between "try again" and "stop, this will never work" — and building code that actually knows it too.

---

Why this matters in data engineering

Think about every external thing your pipeline depends on. An API. A database. A message queue. None of them are perfectly reliable. Things time out. Servers get overloaded. Connections drop.

Some of these failures are temporary — ask again in a moment, and it works. Others are permanent — the request itself was wrong, and asking again changes nothing.

Ask yourself: if your code can't tell these two apart, what happens when a permanent failure hits your pipeline? If your instinct is "it just keeps failing forever, wasting time" — you're right, and that's the small version of the problem. The bigger version is what happens when thousands of instances of your code do this at once, all hitting the same struggling service, all making it worse.

---

The core idea: not all failures deserve a retry

Retryable vs. non-retryable

A failure is retryable if trying again, with no other changes, might actually succeed. A timeout. A "server temporarily unavailable." A dropped connection. These are about the moment, not the request.

A failure is non-retryable if the request itself was wrong. An invalid input. A malformed field. Something that will be exactly as wrong the second time, the tenth time, the hundredth time.

Ask yourself this question about any failure before deciding to retry it: if I send the exact same request again, right now, with nothing changed, could this possibly succeed? If the honest answer is no, retrying isn't caution — it's just delay before an outcome that was already decided.

Backoff: waiting longer each time

If something is genuinely retryable, retrying instantly, over and over, is still a bad idea. Backoff means waiting a little longer before each retry — often doubling the wait each time. This gives whatever you're calling a real chance to recover, instead of hitting it again the instant it's already struggling.

Jitter: not everyone waiting the same amount

Here's a problem backoff alone doesn't solve. Imagine a thousand different clients all hit the same failure at the same moment. If they all back off on the exact same schedule, they'll all retry at the exact same moment too — hitting the struggling service in synchronized waves, which can be worse than if they'd never backed off at all.

Jitter means adding a bit of randomness to the wait time, so retries spread out instead of arriving all at once. This is a small detail with an outsized effect at real scale.

A classified exception hierarchy

None of the above works if your code can't actually tell a retryable failure from a non-retryable one. That means your exceptions need to say what kind of failure they represent — not just "something broke," but "this was temporary" or "this was permanent." Once your exceptions carry that information, your retry logic can make a real decision instead of guessing.

The dead-letter path

What happens to a record that fails, and isn't going to succeed no matter what? It shouldn't just vanish. It shouldn't crash your whole pipeline over one bad record either. It should go somewhere — a dead-letter path — where you can come back later and actually look at what failed, and why.

---

Manual lab: don't retry what will never succeed

Getting the lab files

Download module-3-materials.zip from the course platform, same as before:

bashCopy
cd ~/courses/pp4de
cp /mnt/c/Users/yourname/Downloads/module-3-materials.zip course-materials/
cd course-materials
unzip module-3-materials.zip

You should now have course-materials/module-3/starter/ and course-materials/module-3/solution/. Commit it:

bashCopy
cd ~/courses/pp4de
git add course-materials/module-3
git commit -m "Add Module 3 lab materials"
git push

The scenario

You've inherited a small billing tool. Every day, it needs to fetch a handful of currency exchange rates from a third-party FX service, so it can convert charges into the right currencies. The service is a little flaky — sometimes it's briefly unavailable, sometimes it times out — but it usually recovers within a couple of seconds.

One day, someone adds a new currency pair to the billing run. It's a typo — USD/XXX isn't a real pair. The job doesn't fail cleanly. It just... takes a lot longer than usual.

Reproduce the problem first

1. Go into the starter folder:

   bashCopy
   cd course-materials/module-3/starter

2. Run it:

   bashCopy
   python3 run.py

3. Watch the output closely. You'll see USD/EUR and USD/GBP fail a couple of times each, then succeed — that part is fine, those are genuinely temporary problems, and retrying them worked.
4. Now look at what happens with USD/XXX. Count how many times it says attempt N failed. All five. Every single one, with the exact same error, every time. Read fx_client.py — this pair was never going to succeed, not on attempt 1, not on attempt 5.
5. Open rate_fetcher.py. Find the line that catches the failure. Look at what it catches — a single broad exception type that covers everything, including the pair that was never going to work. Nothing in this code can tell the difference between "try again" and "don't bother."
6. One more thing worth noticing: look at the delay between each attempt. It's identical every time — no backoff, no variation at all.

You've now watched, directly, what this module exists to fix: a real request that could never succeed, retried exactly as many times as one that genuinely could.

Your task

Fix this so that:

1. Retryable failures (timeouts, temporary unavailability) still get retried — with real backoff and jitter, not a fixed delay.
2. Non-retryable failures (an invalid currency pair) fail immediately, on the first attempt — no wasted retries at all.
3. A failure that's genuinely exhausted its retries, or was never retryable to begin with, gets recorded somewhere — a dead-letter list — instead of crashing the whole run.
4. The rest of the billing run keeps going even if one pair fails completely.

A question worth asking before you look at the solution

Same habit as the last two modules: if the solution looks different from the starter, ask why, don't just assume it should.

Here's the honest answer. The shape of "call the service, handle a failure" is the same idea throughout — nothing about that changes. What's different is that the starter's version can't distinguish which kind of failure it's looking at, so it treats all of them the same way. The solution's version does exactly one new thing structurally: it separates "worth retrying" from "not worth retrying," and gives each path somewhere sensible to go. That's not a rewrite of the concept — it's the concept this module is teaching, applied to code that didn't have it yet.

Full worked solution

The complete solution lives in course-materials/module-3/solution/:

solution/
├── pyproject.toml
├── run.py
└── src/
    └── fxrates/
        ├── __init__.py
        ├── exceptions.py     # the classification: what's retryable, what isn't
        ├── fx_client.py       # the simulated third-party service (unchanged)
        ├── retry.py            # backoff + jitter, retrying only what's classified retryable
        ├── dead_letter.py      # where failures go instead of vanishing or crashing the run
        └── rate_fetcher.py     # thin wrapper connecting the client to the retry helper

exceptions.py — the classification itself:

pythonCopy
class RateServiceError(Exception):
    """Base class for anything that can go wrong calling the FX service."""


class RateServiceTimeout(RateServiceError):
    """The service didn't respond in time. Transient - retrying can help."""


class RateServiceUnavailable(RateServiceError):
    """The service is temporarily down. Transient - retrying can help."""


class InvalidCurrencyPairError(RateServiceError):
    """The requested pair doesn't exist. Permanent - retrying never helps."""

Notice this file doesn't do anything on its own — it just gives every kind of failure an honest name. That naming is what makes the next file possible.

retry.py — backoff, jitter, and the actual classification decision:

pythonCopy
import random
import time
from typing import Callable, TypeVar

from fxrates.exceptions import RateServiceTimeout, RateServiceUnavailable

T = TypeVar("T")

# Only these two are worth retrying. Anything else - like
# InvalidCurrencyPairError - is a permanent failure, and isn't listed
# here on purpose, so it propagates immediately instead of being caught.
RETRYABLE_EXCEPTIONS = (RateServiceTimeout, RateServiceUnavailable)


def call_with_retry(
    func: Callable[[], T],
    *,
    max_attempts: int = 5,
    base_delay: float = 0.05,
    max_delay: float = 2.0,
) -> T:
    attempt = 0
    while True:
        attempt += 1
        try:
            return func()
        except RETRYABLE_EXCEPTIONS as e:
            if attempt >= max_attempts:
                raise
            backoff = min(max_delay, base_delay * (2 ** (attempt - 1)))
            delay = random.uniform(0, backoff)
            print(f"    attempt {attempt} failed ({e}); retrying in {delay:.3f}s")
            time.sleep(delay)

The single most important line in this whole module is RETRYABLE_EXCEPTIONS = (RateServiceTimeout, RateServiceUnavailable). InvalidCurrencyPairError deliberately isn't in that tuple — which means except RETRYABLE_EXCEPTIONS simply doesn't catch it at all. It propagates immediately, on the first attempt, with zero retries wasted. That's not an accident of the code; it's the entire fix, expressed in one line.

dead_letter.py — a place for failures to go:

pythonCopy
from dataclasses import dataclass, field


@dataclass
class DeadLetter:
    entries: list[dict[str, str]] = field(default_factory=list)

    def add(self, identifier: str, reason: str) -> None:
        self.entries.append({"identifier": identifier, "reason": reason})

    def __len__(self) -> int:
        return len(self.entries)

rate_fetcher.py — connects the two:

pythonCopy
from fxrates.fx_client import fetch_rate
from fxrates.retry import call_with_retry


def get_rate(pair: str) -> float:
    return call_with_retry(lambda: fetch_rate(pair))

Verify it, step by step

1. Create a virtual environment and install:

   bashCopy
   cd course-materials/module-3/solution
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -e .
   pip install mypy

2. Run mypy --strict, and confirm it passes cleanly:

   bashCopy
   python -m mypy --strict src/fxrates/ run.py

3. Run it:

   bashCopy
   python run.py

   Watch closely. USD/EUR and USD/GBP should still retry and succeed — but this time, notice the delay before each retry is different every time you run it. That's jitter, actually working.
4. Now look at USD/XXX. It should fail once — a single line — and go straight to the dead-letter summary at the bottom. No retries at all. Compare this directly to what you watched the starter code do: five wasted attempts, identical delay, no distinction.
5. Read the final summary printed at the end. You should see how many rates succeeded, and a list of what got dead-lettered and why. The whole run completed — one bad pair didn't take down the other three.

If step 4 showed you a single failed attempt instead of five, and step 5 showed you a clean summary instead of a crash, you've verified the actual thing this module set out to teach.

---

AI-assisted round

The task

Ask your assistant to extend the solution with a new capability:

Output / Note

"Add a get_historical_rate(pair, date) function to this project, that fetches a historical exchange rate from the same flaky service and needs the same kind of retry protection as get_rate already has. Write a test for it, and run it before telling me you're done."

The known failure pattern to watch for

This is the single highest-value catch in this entire course, worth treating that seriously. The pattern: an assistant extending retry logic often writes a new, separate retry loop from scratch, instead of reusing the classification you already built — and that new loop frequently catches a broad exception type again, undoing the exact fix this module is about.

Here's what that tends to look like:

diffCopy
+ def get_historical_rate(pair: str, date: str) -> float:
+     max_attempts = 5
+     for attempt in range(max_attempts):
+         try:
+             return fetch_historical_rate(pair, date)
+         except RateServiceError as e:
+             wait = 2 ** attempt
+             time.sleep(wait)
+     raise RuntimeError(f"Failed after {max_attempts} attempts")

Two separate problems here, both worth catching:

1. It catches RateServiceError, the broad base class — not the RETRYABLE_EXCEPTIONS tuple you already defined. That base class includes InvalidCurrencyPairError. This one new function has quietly reintroduced the exact bug the rest of this module fixed, right next to code that already got it right.
2. No jitter — wait = 2 ** attempt is pure exponential backoff, with no randomness at all. Every caller retrying after the same failure backs off on the identical schedule.

The subtle danger here: this code runs, looks reasonable, and even "works" most of the time — because most of the calls it makes probably succeed on a retryable failure. The bug only shows up the first time someone calls it with a genuinely invalid pair, and by then, it's already shipped.

The guardrail

If you catch this, write it down:

Output / Note

Module 3 guardrail: When an AI assistant adds new code that needs retry logic, check whether it reused the existing classified retry helper — or wrote a new loop that catches a broad exception type again. A second retry implementation is a second chance to reintroduce the exact bug the first one already fixed.

If your assistant correctly reused call_with_retry and RETRYABLE_EXCEPTIONS without being told to — that's still worth logging. Note what you checked and confirmed clean.

---

Common mistakes

• Catching a broad exception type "to be safe." The wider the except clause, the more likely it silently includes something that should never be retried. Narrow, deliberate exception types are safer than broad ones here, not more fragile.
• Backoff with no jitter. Works fine in testing, with one caller. Fails badly at real scale, with thousands of callers retrying in lockstep.
• Treating "it eventually succeeded" as proof the retry logic is correct. A retry loop that wastes five attempts on something unretryable, then happens to succeed on a genuinely transient failure elsewhere, can look totally fine in a quick test — the waste is invisible unless you're specifically watching for it.
• Letting one bad record take down an entire run. If a single unretryable failure crashes the whole job, every other record — including the ones that were completely fine — pays the price too.

---

Capstone tie-in

Step 1: Open your capstone repo and start the infrastructure

Confirm the title bar says pp4de [WSL: Ubuntu].

Open the integrated terminal (Ctrl+`) and run:

bashCopy
docker compose start
docker compose ps

Confirm both services are healthy before continuing — you'll need the mock API's real flakiness for this module's work.

Step 2: Build the exception hierarchy for your capstone

In VS Code's Explorer:

1. Right-click pipeline (inside src/) → New File → name it exceptions.py.

2. Write a classified hierarchy for your capstone's actual failure modes. Base it on what the mock API can genuinely do — you already know this from Module 0 and from the infrastructure itself:

   • A timeout (the API can be told to hang, via force_timeout or natural TIMEOUT_RATE)
   • A rate limit response, 429 (via force_429 or natural rate limiting)
   • A malformed record that fails your Module 2 pydantic model — this one is not retryable. The record was wrong; retrying the same fetch returns the same wrong record.

   At minimum, something like:

   pythonCopy
   class IngestionError(Exception):
       """Base class for anything that can go wrong ingesting a record."""
   
   
   class UpstreamTimeout(IngestionError):
       """The API didn't respond in time. Retryable."""
   
   
   class UpstreamRateLimited(IngestionError):
       """The API returned 429. Retryable, after waiting."""
   
   
   class RecordValidationError(IngestionError):
       """A record failed validation. Not retryable - the data is wrong."""

Step 3: Build your retry helper

In VS Code's Explorer:

1. Right-click pipeline → New File → name it retry.py.
2. Adapt the same pattern from this module's lab — a classified RETRYABLE_EXCEPTIONS tuple, real backoff, real jitter. Reuse the actual code shape from course-materials/module-3/solution/src/fxrates/retry.py rather than rewriting it from scratch; the logic doesn't need to be different here, just applied to your capstone's own exceptions.

Step 4: Prove it against the real mock API — the one mandatory terminal step

Open the integrated terminal, confirm your venv is active, and try hitting the mock API's deliberate failure modes directly:

bashCopy
python3 -c "
import urllib.request

try:
    urllib.request.urlopen('http://localhost:8000/records?date=2026-07-01&page=1&force_timeout=true', timeout=2)
except Exception as e:
    print('Got:', type(e).__name__, e)
"

You should see a timeout error. This confirms the mock API's force_timeout flag genuinely produces the failure your retry logic needs to handle — don't just assume it does, watch it happen.

Step 5: Commit and push through VS Code's Source Control panel

1. Click the Source Control icon in the sidebar.
2. Confirm .venv is not listed under Changes.
3. You should see exceptions.py and retry.py listed under Changes, inside src/pipeline/.
4. Stage, commit with a message like Module 3: classified exceptions, backoff and jitter retry helper, and click Commit.
5. Click Sync Changes to push to GitHub.
6. Confirm the changes appear on GitHub.

Check row 3 off your SPEC.md checklist. Six to go.

Before you close for the day:

bashCopy
docker compose stop

---

Interview drill

Every question below follows the same pattern. First, the question. Then, what the interviewer wants to hear. Then, a junior engineer's answer. Then, a strong senior answer.

Recall

Question: "What's the actual difference between backoff and jitter? Why do you need both, not just one?"

What the interviewer wants to hear: a clear, separate explanation of each, and a specific reason jitter matters that goes beyond "it adds randomness" — ideally tied to what happens at real scale, with many callers.

Junior answer: "Backoff means waiting longer between retries. Jitter adds some randomness to that wait so it's not exactly the same every time."

(Correct, but doesn't explain why the randomness actually matters — sounds like a memorized definition, not something understood.)

Senior answer: "Backoff is about giving a struggling service time to recover — each retry waits longer than the last, usually doubling. On its own, though, backoff has a hidden problem: if a thousand different clients all hit the same failure at the same moment, and they all back off on the same schedule, they all retry again at the same moment too — just in a later, still-synchronized wave. Jitter breaks that synchronization by adding randomness to the wait, so retries spread out over time instead of arriving in lockstep. I've seen this matter for real — a service recovers, then immediately gets hit by a wall of synchronized retries the moment it comes back up, which can knock it back down. Jitter is specifically the fix for that failure mode, not just general noise."

Debugging

Question: shown as a real snippet and incident story, with no explanation yet:

pythonCopy
import time
import requests

def push_reading(reading: dict) -> None:
    max_attempts = 5
    for attempt in range(max_attempts):
        try:
            resp = requests.post(
                "https://telemetry.internal/readings",
                json=reading,
                timeout=3,
            )
            resp.raise_for_status()
            return
        except requests.RequestException:
            wait = 2 ** attempt
            time.sleep(wait)
    raise RuntimeError(f"Failed to push reading after {max_attempts} attempts")

"This function runs on every field device gateway. During a recent incident, a bad deploy caused the telemetry endpoint to reject every request with 400 Bad Request for about 40 minutes — a schema mismatch, not an outage. During that window, thousands of gateways were retrying constantly, and the telemetry service's ingress got so overloaded that other, unrelated traffic on the same load balancer started timing out too. What's wrong with this retry logic, and what part of it turned a 40-minute schema bug into a much wider outage?"

What the interviewer wants to hear: identifying indiscriminate retrying of a non-retryable error as the root cause, and separately naming the missing jitter as a second, compounding problem — not just one or the other.

Junior answer: "The retry logic looks reasonable to me — it backs off exponentially, which is good practice. Maybe the real issue is the telemetry service itself couldn't handle the load."

(Accepts the retry logic at face value because it "looks like" good practice — exponential backoff — without checking what it actually catches.)

Senior answer: "except requests.RequestException catches everything — including HTTPError from raise_for_status(), for any status code, 400 included. A 400 means the request itself was malformed; retrying it with no changes will never succeed. This code can't tell that apart from a genuine timeout or a 5xx, so it retried a permanently broken request exactly as hard as it would retry a temporary one. That's the root cause — thousands of gateways hammering an endpoint that was never going to accept their requests. The missing jitter makes it worse on top of that: every gateway backs off on the same exponential schedule, so their retries stay synchronized instead of spreading out, hitting the server in correlated waves. The fix is two things: classify exceptions so 4xx fails fast instead of retrying, and add jitter so any retries that do happen aren't synchronized across the whole fleet."

AI-review

Question: shown this diff, with no explanation yet:

diffCopy
+ def get_historical_rate(pair: str, date: str) -> float:
+     max_attempts = 5
+     for attempt in range(max_attempts):
+         try:
+             return fetch_historical_rate(pair, date)
+         except RateServiceError as e:
+             wait = 2 ** attempt
+             time.sleep(wait)
+     raise RuntimeError(f"Failed after {max_attempts} attempts")

The project already has a call_with_retry helper with a classified RETRYABLE_EXCEPTIONS tuple, used by every other function that calls the FX service.

What the interviewer wants to hear: recognizing this as the exact same bug the rest of the project already solved, reintroduced in one new function that didn't reuse the existing fix.

Junior answer: "It has its own retry loop with backoff, so it should be fine — looks like reasonable error handling to me."

(Sees a retry loop with backoff and assumes that's sufficient, without checking what exception type it actually catches.)

Senior answer: "This function wrote its own retry loop from scratch instead of reusing call_with_retry, and in doing that, it brought back the exact bug the rest of the project already fixed — it catches RateServiceError, the broad base class, which includes InvalidCurrencyPairError. That means an invalid pair passed to this function will get retried five times for no reason, right next to code that correctly fails fast on the same kind of error. It's also missing jitter — wait = 2 ** attempt is pure exponential backoff with no randomness. I'd tell the assistant to delete this function's custom retry loop entirely and call call_with_retry instead, the same way get_rate does — there's no reason for this project to have two different, inconsistent retry implementations."

Judgment

Interviewer: "You're the on-call engineer for an IoT platform ingesting sensor readings from field devices. A vendor firmware update just shipped, and it introduced a bug — a small percentage of devices are now sending a malformed reading that your ingestion service rejects with a validation error. Your retry logic currently retries every failure the same way, five times with backoff. Your manager asks you to just increase the retry count to make sure readings eventually get through. What do you say?"

Candidate (asking first): "Before I answer — do we know roughly what percentage of devices are affected, and is the malformed data something we can even parse correctly once, or is it just wrong?"

Interviewer: "Good question. Roughly 8% of devices are affected. The data itself is genuinely wrong — a firmware bug is putting the temperature reading in the wrong field. It's not a formatting quirk we could work around, it's actually incorrect."

Candidate (first answer): "Then I wouldn't increase the retry count at all — I'd push back on that specific idea. If the data is genuinely wrong, not just slow to arrive, retrying it five times or five hundred times changes nothing. The reading will fail validation every single time, because the underlying problem is in the firmware, not in a timing issue our retries could paper over."

Interviewer (pushback): "But readings are being lost right now. Doesn't more retries at least give us a better chance some of them get through eventually?"

Candidate (round 1): "I'd separate two different problems here. Some of what's failing might genuinely be transient — a timeout, a brief service hiccup — and for that, our existing retry logic already does its job. But for the 8% of devices sending malformed data, more retries doesn't improve their odds at all, because the problem isn't random, it's deterministic — wrong firmware, wrong field, every single time. Increasing the retry count for everyone would just mean we retry the genuinely broken readings for longer, wasting time and load, without actually recovering a single one of them."

Interviewer (second pushback): "So what would you actually do instead, given we can't ship a firmware fix instantly?"

Candidate (round 2): "I'd make sure those malformed readings are correctly classified as non-retryable, so they fail fast and go straight to a dead-letter path instead of consuming retry attempts. That gets them out of the retry loop entirely — no more wasted load on our ingestion service from a problem retries can't fix. Then, separately, I'd want visibility into that dead-letter path — a count of how many readings are landing there, broken down by device, so we can actually see the scope of the firmware issue and follow up with the vendor with real numbers, instead of just quietly losing data."

Interviewer (final challenge): "Isn't dead-lettering these readings still just losing data, the same as what's happening now?"

Candidate (final defense): "There's a real difference between silently losing data and deliberately capturing it somewhere inspectable. Right now, if I understand it, these readings are consuming five retry attempts each before eventually failing anyway — so we're already losing them, just more expensively and less visibly. Dead-lettering them means we stop pretending they'll succeed, we free up retry capacity for readings that can actually be recovered, and we get a concrete, countable record of exactly what's failing and why — which is what actually gets a vendor to prioritize a firmware fix. Increasing the retry count doesn't get any of that. It just makes the existing waste larger."

---

Model strong answer — a single answer, given upfront:

Output / Note

"I'd push back on increasing the retry count, and explain why before proposing an alternative. First, I'd want to know whether the failing readings are genuinely transient or deterministically wrong — in this case, a firmware bug is putting data in the wrong field, so it's wrong every time, not occasionally slow. Retrying a deterministic failure more times doesn't improve the odds at all; it just spends more time and load arriving at the same failure. Our existing retry logic already handles genuinely transient issues fine. What I'd actually do is make sure these malformed readings are classified as non-retryable, so they fail fast and go straight to a dead-letter path instead of eating five retry attempts each for nothing. Then I'd add visibility into that dead-letter path — a count, broken down by device — so we have concrete numbers to bring to the firmware vendor, instead of just quietly losing data more expensively than we need to. That's a real fix aimed at the actual problem. Increasing the retry count isn't a fix at all — it's spending more resources to arrive at the exact same failure, just slower."