Python Foundation for Data Engineers

Chapter 7 — Handling Errors Gracefully

Lesson 7.1 — Reading a Python Error Like a Detective

Back in Chapter 1, you broke your very first program on purpose, and saw a wall of red text appear. You moved past it quickly then, just to prove the run-fix-rerun loop worked. Now let's actually slow down and learn to read that red text properly, because it's one of the most useful skills in this entire course.

Let's cause one on purpose, with something you've written many times by now:

python
row_count = "4500"
print(row_count + 1)
Output / Note

Traceback (most recent call last): File "script.py", line 2, in print(row_count + 1) TypeError: can only concatenate str (not "int") to str

This block is called a traceback, and it's genuinely trying to help you, not scold you. Read it from the bottom up — that's the order that actually makes sense.

The very last line is the headline: TypeError: can only concatenate str (not "int") to str. That tells you two things immediately — the category of error, TypeError, meaning you used a value in a way its type doesn't support, and the specific problem, that Python tried to combine text and a number and didn't know how.

The line above that, print(row_count + 1), shows you exactly which line of your code caused it. And the line above that tells you which file and line number, useful once your scripts span more than one file.

So the full story, read properly: "On line 2, you tried to combine a string and an int, and that's not allowed." That's not a cryptic puzzle. That's a precise bug report, handed to you for free, every single time something goes wrong.

Here's a second common one, worth recognizing on sight:

python
customer = {"name": "Priya Shah"}
print(customer["email"])
Output / Note

Traceback (most recent call last): File "script.py", line 2, in print(customer["email"]) KeyError: 'email'

A KeyError means you tried to look up a dictionary key that doesn't exist — exactly the situation .get(), from Chapter 4, was built to handle safely. Recognizing the error type by name, before you even read the details, will get faster the more of them you see. That's not a sign you're doing something wrong — it's just what building real software actually looks like, for every engineer, every day.

Try It Yourself: Deliberately write a line of code that causes an IndexError — try to access an index in a list that doesn't exist, like my_list[10] on a list with only 3 items. Read the traceback from the bottom up, and in your own words, write one sentence explaining what it's telling you.

Lesson 7.2 — try / except / finally

Reading an error after your program has already crashed is useful. But often, you don't want your program to crash at all — you want it to notice a problem, handle it gracefully, and keep going. That's what try and except are for.

python
row_count = "4500"

try:
    print(row_count + 1)
except TypeError:
    print("Couldn't add — row_count wasn't a number")
Output / Note

Couldn't add — row_count wasn't a number

Read it like a sentence: "try to run this code — but if a TypeError happens, run this other code instead, rather than crashing." Notice your program kept running afterward, instead of stopping dead with a traceback. That's the entire point — you've turned a crash into a handled, expected situation.

You can catch different error types differently, which matters because different problems often need different responses:

python
def safe_divide(total, count):
    try:
        return total / count
    except ZeroDivisionError:
        print("Can't divide by zero — returning 0 instead")
        return 0

print(safe_divide(4500, 0))
print(safe_divide(4500, 3))
Output / Note

Can't divide by zero — returning 0 instead 0 1500.0

This is a genuinely common real situation — calculating an average error rate, for example, where the total row count might legitimately be zero on a day a file didn't arrive at all. Instead of your whole pipeline crashing over that one edge case, it handles it, reports it clearly, and moves on.

One more piece worth knowing: finally. Code inside a finally block runs no matter what — whether the try succeeded or an error was caught.

python
try:
    print("Processing file...")
    result = 100 / 0
except ZeroDivisionError:
    print("Error: division by zero")
finally:
    print("File processing attempt complete")
Output / Note

Processing file... Error: division by zero File processing attempt complete

finally is genuinely useful for cleanup work you always want to happen — like closing a file or a database connection — regardless of whether things went smoothly or not. You won't need it constantly in this course, but it's worth recognizing when you see it.

A word of honest caution: don't catch errors just to silence them. except Exception: pass — catching everything and doing nothing — hides real problems instead of solving them, and it's one of the most common bad habits in beginner code. Catch specific, expected error types, and always do something meaningful in response, even if that "something" is just a clear message about what went wrong.

Try It Yourself: Write a function safe_get_price(order, key) that tries to return order[key], and catches a KeyError by returning 0 instead, along with a printed message saying which key was missing. Test it on a dictionary that's missing the key you ask for.

Lesson 7.3 — Raising Your Own Errors on Purpose

So far, every error you've seen has come from Python itself, catching a mistake automatically. But sometimes, you know something is wrong before Python does — and the right move is to stop things deliberately, on your own terms, rather than let bad data quietly continue further into a pipeline.

That's what raise is for.

python
def load_price(price):
    if price < 0:
        raise ValueError(f"Price cannot be negative: {price}")
    return price

print(load_price(24.99))
print(load_price(-5))
Output / Note

24.99 Traceback (most recent call last): File "script.py", line 6, in print(load_price(-5)) File "script.py", line 3, in load_price raise ValueError(f"Price cannot be negative: {price}") ValueError: Price cannot be negative: -5

Notice the first call worked fine and printed 24.99. The second one hit our own raise line, and stopped the program with a clear, specific message — one we wrote ourselves, in plain language, describing exactly what went wrong and what the bad value actually was.

Here's the honest reasoning behind this, because it can feel backwards at first: why would you want your own code to crash on purpose? Because a loud, clear, immediate failure is almost always better than a quiet, wrong one. A negative price that silently slides into a revenue report will cause real, confusing damage days or weeks later, to someone who has no idea where it came from. A raise, right at the source, stops it in its tracks, at the exact moment and place the bad data was found.

You can combine raise with try/except, catching your own deliberate errors just like Python's built-in ones:

python
def load_price(price):
    if price < 0:
        raise ValueError(f"Price cannot be negative: {price}")
    return price

prices = [24.99, -5, 15.50]

for price in prices:
    try:
        print(f"Loaded: {load_price(price)}")
    except ValueError as error:
        print(f"Skipped bad row: {error}")
Output / Note

Loaded: 24.99 Skipped bad row: Price cannot be negative: -5 Loaded: 15.50

That as error bit captures the actual error message, so you can print it, log it, or include it in a report. Look at what just happened: a batch of three prices, one clearly bad, and the loop handled it cleanly — skipping only the broken row, reporting exactly why, and still processing the two good ones. That's a very real, very common shape for a pipeline to take.

Try It Yourself: Write a function check_row_count(count) that raises a ValueError with a clear message if count is 0, and otherwise returns the count. Then write a loop over [4500, 0, 3200] that calls this function inside a try/except, printing either the valid count or a "skipped" message for the bad one.

Lesson 7.4 — Hands-On: Handle a Missing or Broken File

Let's bring this chapter together on a task every data engineer eventually hits, usually on their very first week: a file that's supposed to be there simply isn't, or a value inside it isn't what you expected.

Here's a function that reads a row count out of a dictionary — the kind of small piece you'd find inside a bigger pipeline script:

python
def get_row_count(file_info):
    return file_info["rows"]

Simple enough, until it's handed something unexpected:

python
files = [
    {"name": "orders.csv", "rows": 4500},
    {"name": "customers.csv"},
    {"name": "billing.csv", "rows": "unknown"},
]

for file_info in files:
    print(get_row_count(file_info))

Run this, and it crashes entirely on the second file — no "rows" key at all — and never even gets to the third one. One bad file just took down the whole batch. That's exactly the fragile behavior we're about to fix.

Your task:

  1. Wrap the row count lookup in a try/except that catches a missing "rows" key.
  2. Also handle the case where "rows" exists but isn't a proper number, like "unknown".
  3. For any problem file, print a clear message naming the file and what was wrong — but keep processing the rest of the batch.
  4. Count how many files loaded successfully versus how many were skipped.

Here's the shape to build from:

python
files = [
    {"name": "orders.csv", "rows": 4500},
    {"name": "customers.csv"},
    {"name": "billing.csv", "rows": "unknown"},
    {"name": "inventory.csv", "rows": 980},
]

loaded = 0
skipped = 0

for file_info in files:
    name = file_info.get("name", "Unnamed file")
    try:
        rows = file_info["rows"]
        rows = int(rows)
        print(f"{name}: {rows} rows loaded")
        loaded += 1
    except KeyError:
        print(f"{name}: skipped — no row count found")
        skipped += 1
    except ValueError:
        print(f"{name}: skipped — row count wasn't a valid number")
        skipped += 1

print(f"\nDone. {loaded} file(s) loaded, {skipped} skipped.")
Output / Note

orders.csv: 4500 rows loaded customers.csv: skipped — no row count found billing.csv: skipped — row count wasn't a valid number inventory.csv: 980 rows loaded

Done. 2 file(s) loaded, 2 skipped.

Look at what changed. The exact same messy batch that completely crashed the first version now processes cleanly end to end — every good file loaded, every bad one clearly flagged with a specific, honest reason, and a summary count at the end that tells you at a glance whether this morning's run needs attention.

This is genuinely the difference between a script and a pipeline. A script works when the input is clean. A pipeline is expected to survive the input being messy — because in this job, it always eventually is.

Try It Yourself: Add a fifth file to the list with "rows": -50. Add a new check inside the try block that raises your own ValueError if the row count is negative, with a clear message — and confirm your existing except ValueError handling catches it correctly, without any extra code changes.