Python Foundation for Data Engineers

Chapter 22 — Building Resilient API Clients

Lesson 22.1 — Timeouts: Don't Wait Forever

Here's a genuinely uncomfortable fact about the internet: sometimes, a server doesn't respond quickly, and doesn't respond slowly either — it just never responds at all. No error, no 500, nothing. Your code simply sits there, waiting, forever, unless you tell it not to.

By default, requests will wait indefinitely for a response. Let's see the fix — a timeout, telling requests exactly how long it's allowed to wait before giving up and raising an error instead.

python
import requests

try:
    response = requests.get("https://httpbin.org/delay/10", timeout=3)
    print(response.status_code)
except requests.exceptions.Timeout:
    print("The request took too long — giving up after 3 seconds")
Output / Note

The request took too long — giving up after 3 seconds

That httpbin.org/delay/10 URL is another practice tool — it deliberately waits 10 seconds before responding, on purpose, so you can see a timeout happen safely, on demand. Notice timeout=3 — we told requests to wait at most 3 seconds. Since the server was going to take 10, requests gave up on its own, and raised a requests.exceptions.Timeout — an error you can catch, exactly the way you caught ZeroDivisionError and KeyError back in Week 1, Chapter 7.

Here's the honest, practical reasoning: without a timeout, one slow or unresponsive server can freeze your entire pipeline indefinitely — no crash, no error message, just a script that silently never finishes, and a confused engineer wondering why nothing's happening. A timeout turns that silent, indefinite hang into a clear, specific, catchable error — exactly the same philosophy from Week 1, Chapter 7's raise lesson: a loud, honest failure beats a quiet, endless one.

The practical habit to build starting right now: every single requests.get() call you write from this point forward should include a timeout. A few seconds is typical for most APIs; adjust based on what you know about how responsive a given API tends to be.

Lesson 22.2 — Retries with Backoff

Back in Week 1, Chapter 5.3, you built a small retry loop — trying to "connect" a few times before giving up. That was a genuinely good first attempt at an idea you're about to build properly, for real, with a real network call behind it.

Here's the honest problem retries solve: plenty of API failures are temporary. A server hiccups for half a second, a network blip drops one request — and trying again, moments later, often just works. Giving up completely after a single failed attempt throws away data unnecessarily, for a problem that may have already resolved itself.

Let's build this properly:

python
import requests
import time

def get_with_retries(url, max_attempts=3, timeout=5):
    """
    Make a GET request, retrying on failure up to max_attempts times.
    Waits longer between each retry (exponential backoff).
    Returns the response if successful, or None if every attempt fails.
    """
    for attempt in range(1, max_attempts + 1):
        try:
            response = requests.get(url, timeout=timeout)
            if response.status_code == 200:
                return response
            print(f"Attempt {attempt}: got status {response.status_code}")
        except requests.exceptions.RequestException as error:
            print(f"Attempt {attempt}: request failed — {error}")

        if attempt < max_attempts:
            wait_time = 2 ** attempt
            print(f"Waiting {wait_time} seconds before retrying...")
            time.sleep(wait_time)

    print("All attempts failed")
    return None

Let's break down the new idea here: wait_time = 2 ** attempt. On attempt 1, that's 2 ** 1, or 2 seconds. On attempt 2, 2 ** 2, or 4 seconds. On attempt 3, 2 ** 3, or 8 seconds. This growing pause is called exponential backoff, and it's genuinely the standard, respectful approach used across real-world systems: wait a little longer after each failure, rather than hammering an already-struggling server with requests at a constant, aggressive pace.

Notice requests.exceptions.RequestException — this is a genuinely useful detail. It's not one specific error, but a broad category that covers timeouts, connection failures, and several other network problems all at once, similar to how you might catch a general category of error rather than every specific type individually. This means our function survives several different kinds of real-world network trouble, not just one.

Let's test it against a URL that's guaranteed to fail, to see the honest, patient retry behavior in action:

python
result = get_with_retries("https://httpbin.org/status/500", max_attempts=3)
print(result)
Output / Note

Attempt 1: got status 500 Waiting 2 seconds before retrying... Attempt 2: got status 500 Waiting 4 seconds before retrying... Attempt 3: got status 500 All attempts failed None

Three honest attempts, a growing pause between each, and a clear, final admission of failure — None — rather than crashing your entire pipeline over one troublesome request. Whatever called this function now has to decide, deliberately, what to do about a None result — exactly the same honest-failure-handling instinct from Week 1, Chapter 7.4's messy file exercise.

Lesson 22.3 — Handling Partial & Failed Responses Gracefully

A retry loop handles one kind of problem: a request that fails outright. But there's a second, sneakier kind of problem worth guarding against — a request that "succeeds," with a 200 status code, but hands back something you weren't actually expecting: a malformed JSON body, or a response missing a field your code assumes will be there.

Let's see the malformed-JSON case first:

python
import requests

response = requests.get("https://httpbin.org/html")

try:
    data = response.json()
except requests.exceptions.JSONDecodeError:
    print("Response wasn't valid JSON — can't parse it")
Output / Note

Response wasn't valid JSON — can't parse it

That URL deliberately returns an HTML page, not JSON — and calling .json() on it raises requests.exceptions.JSONDecodeError, exactly the same kind of specific, catchable error from Week 1, Chapter 7. A 200 status told us the request succeeded — but it told us nothing about whether the content was actually what we expected. Checking the status code alone isn't enough; the content itself needs its own honest check too.

The second sneaky case — a field you expected simply isn't there:

python
data = {"id": 101, "name": "Priya Shah"}

email = data.get("email")
if email is None:
    print("Warning: this record is missing an email field")
else:
    print(f"Email: {email}")
Output / Note

Warning: this record is missing an email field

Recognize .get()? Exactly the safe-lookup tool from Week 1, Chapter 4, now doing genuinely important work — protecting your code from a KeyError crash the moment a real API sends back a record shaped slightly differently than you assumed it always would. Real APIs do this more often than you'd expect: a field that's usually present becomes optional, or a new API version quietly renames something.

Let's combine both defenses into one small, honest helper function:

python
import requests

def safe_parse_response(response):
    """
    Safely parse a response's JSON body.
    Returns the parsed data, or None if the response wasn't valid JSON.
    """
    try:
        return response.json()
    except requests.exceptions.JSONDecodeError:
        print("Warning: response was not valid JSON")
        return None


response = requests.get("https://jsonplaceholder.typicode.com/posts/1")
data = safe_parse_response(response)

if data is not None:
    title = data.get("title", "No title provided")
    print(title)
Output / Note

sunt aut facere repellat provident occaecati excepteur

Notice the layered honesty here, exactly the philosophy this whole course has been building: check the status code, then check the content actually parses correctly, then check the specific field you need is actually present, with a sensible fallback if it isn't. Each layer catches a genuinely different, real way an API interaction can go slightly wrong, without your code ever assuming everything is fine just because nothing has crashed yet.

Lesson 22.4 — Hands-On: Build a Retry-and-Backoff API Client Function

Let's bring this whole chapter together into one properly resilient function — combining timeouts, retries with backoff, and safe response handling, all in a single, genuinely reusable tool.

python
import requests
import time

def resilient_get(url, params=None, max_attempts=3, timeout=5):
    """
    Make a resilient GET request:
    - Applies a timeout so a slow server can't hang forever
    - Retries on failure with exponential backoff
    - Only treats a 200 status with valid JSON as a real success
    - Returns parsed JSON data on success, or None if every attempt fails
    """
    for attempt in range(1, max_attempts + 1):
        try:
            response = requests.get(url, params=params, timeout=timeout)

            if response.status_code == 200:
                try:
                    return response.json()
                except requests.exceptions.JSONDecodeError:
                    print(f"Attempt {attempt}: got 200 but response wasn't valid JSON")
            else:
                print(f"Attempt {attempt}: got status {response.status_code}")

        except requests.exceptions.Timeout:
            print(f"Attempt {attempt}: request timed out")
        except requests.exceptions.RequestException as error:
            print(f"Attempt {attempt}: request failed — {error}")

        if attempt < max_attempts:
            wait_time = 2 ** attempt
            print(f"Waiting {wait_time} seconds before retrying...")
            time.sleep(wait_time)

    print(f"All {max_attempts} attempts failed for {url}")
    return None

Let's test it against a URL that's genuinely, randomly unreliable — this practice endpoint returns either a 500 error or a 200 success at random, giving you a realistic taste of exactly the kind of flaky behavior this whole function was built to survive:

python
result = resilient_get("https://httpbin.org/status/500,500,200", max_attempts=4)

if result is not None:
    print("Success:", result)
else:
    print("Giving up — this data will need to be retried later or flagged for review")
Output / Note

Attempt 1: got status 500 Waiting 2 seconds before retrying... Attempt 2: got status 500 Waiting 4 seconds before retrying... Attempt 3: got status 200 Success: {}

Your exact output will vary run to run, since that endpoint picks its response randomly — sometimes you'll see success on the first attempt, sometimes not until the last. That unpredictability is genuinely the point: real flaky APIs behave exactly this way, and resilient_get handles all of it, patiently, without crashing your pipeline over what often turns out to be a temporary hiccup.

Now let's use it on real, genuinely reliable data, to confirm the happy path still works cleanly:

python
data = resilient_get("https://jsonplaceholder.typicode.com/users/1")
print(data["name"])
Output / Note

Leanne Graham

You've now built a genuinely production-shaped tool — the kind of function that sits at the base of nearly every real data pipeline that talks to an external API. Every lesson from this chapter is folded into it: don't wait forever, don't give up on the first hiccup, and never trust a response until you've actually confirmed it's what you expected.