Python Foundation for Data Engineers

Chapter 9 — Trust but Verify: Intro to Testing

Lesson 9.1 — Why We Test Data Code

Here's a story that happens more often than anyone likes to admit. A data engineer writes a cleaning function. They run it once, look at the output, it looks right, and they move on. Two weeks later, someone in a meeting asks why last month's revenue report is 8% lower than it should be. It takes three days to trace it back to that same cleaning function — it turns out it silently broke on a specific edge case, one that didn't show up in the sample they eyeballed that one time, weeks ago.

That's the problem with "it looks right." Looking at output once tells you it worked that one time, on that one input. It tells you nothing about tomorrow's file, or the one after that, or what happens the day someone's name has an apostrophe in it, or a price field shows up empty.

This is exactly the gap testing closes. A test is a small, honest piece of code that checks your function does the right thing — automatically, every single time, without you having to remember to look, or trust your own eyes.

Here's the thing worth noticing: you've actually already been doing a rough version of this all course. Every time you wrote print() after cleaning something, just to eyeball whether it worked — that instinct was correct. Testing simply takes that same instinct and makes it permanent, automatic, and honest, instead of a one-time glance you have to remember to repeat.

Think back to Lesson 2.4, the tiny data profiler — you printed values before and after converting them, specifically to prove the conversion actually worked. That was the exact seed of testing, and now you're going to grow it into the real thing.

The habit we're building in this lesson is simple to say, and genuinely valuable to actually build: don't just check your code once by eye — write down what "correct" looks like, and let the computer check it for you, every time, forever.

That's what the rest of this chapter is about.

Lesson 9.2 — Your First pytest Test

Let's write an actual test. We'll use a tool called pytest — the standard, most widely used testing tool in Python, and one you'll see used at nearly every company that writes Python professionally.

First, install it. Open your terminal in VS Code and run:

bash
pip install pytest

Now, let's set up two files, side by side, in your python-de-foundations folder. First, a small function to test — call this file cleaning.py:

python
def clean_name(name):
    return name.strip().title()

Nothing new here — this is straight out of Chapter 3. Now, in the same folder, create a second file, called test_cleaning.py:

python
from cleaning import clean_name

def test_clean_name_strips_and_titles():
    result = clean_name("  priya shah  ")
    assert result == "Priya Shah"

A few important details here, worth reading slowly. That first line, from cleaning import clean_name, pulls the function in from the other file, so this test file can actually use it. The function name, test_clean_name_strips_and_titles, starts with test_ — that's not a style choice, it's a requirement. pytest automatically finds and runs any function starting with test_, in any file starting with test_. That's genuinely the entire setup — no configuration, no extra steps.

And that assert line is the heart of the whole thing. assert means "I'm claiming this is true — check it, and stop everything loudly if it isn't." Read it like a sentence: "assert that result equals 'Priya Shah'." If it does, nothing visible happens — a passing test is quiet on purpose. If it doesn't, pytest will tell you, clearly and specifically.

Now run it. In your terminal, inside the folder containing both files:

bash
pytest
Output / Note

test_cleaning.py . [100%] 1 passed in 0.01s

That single dot represents one test, and it passed. Let's break something on purpose, so you can see what a failure actually looks like — change the assertion to something wrong:

python
def test_clean_name_strips_and_titles():
    result = clean_name("  priya shah  ")
    assert result == "priya shah"
Output / Note

FAILED test_cleaning.py::test_clean_name_strips_and_titles AssertionError: assert 'Priya Shah' == 'priya shah'

Look closely at that message — pytest doesn't just say "it failed." It shows you exactly what your code actually produced, right next to what you claimed it should be. That's an enormous head start on fixing the problem, and it's exactly why reading test failures calmly, the same way you learned to read tracebacks back in Chapter 7, is a skill worth building properly from the start.

Fix the assertion back, run pytest again, and watch it pass.

Try It Yourself: Add a second test function to test_cleaning.py, called test_clean_name_handles_already_clean_input, that checks clean_name("Priya Shah") still correctly returns "Priya Shah" — proving the function doesn't break on input that was already clean. Run pytest and confirm both tests pass.

Lesson 9.3 — Testing a Function: Input In, Answer Out

Now that you've written one test, let's build the habit properly. The core pattern behind almost every test you'll ever write is refreshingly simple: give the function a known input, and check that you get the exact answer you expect back out.

Let's test the email validator you built back in Chapter 6. Add it to cleaning.py:

python
import re

def clean_name(name):
    return name.strip().title()

def is_valid_email(email):
    email = email.strip().lower()
    has_valid_pattern = bool(re.search(r"[\w.]+@[\w.]+\.\w+", email))
    has_one_at_symbol = email.count("@") == 1
    return has_valid_pattern and has_one_at_symbol

Now let's test it properly — and here's an important habit to build early: test both the case that should work, and the case that shouldn't. A test suite that only checks good input is only telling you half the story.

python
from cleaning import clean_name, is_valid_email

def test_clean_name_strips_and_titles():
    assert clean_name("  priya shah  ") == "Priya Shah"

def test_is_valid_email_accepts_good_email():
    assert is_valid_email("priya.shah@company.com") == True

def test_is_valid_email_rejects_double_at_symbol():
    assert is_valid_email("amit.verma@@company.com") == False

def test_is_valid_email_rejects_missing_at_symbol():
    assert is_valid_email("not-an-email") == False
Output / Note

test_cleaning.py .... [100%] 4 passed in 0.01s

Four small, focused tests, each checking exactly one thing. Notice the naming pattern: each test name describes precisely what it's checking, almost like a sentence — test_is_valid_email_rejects_double_at_symbol tells you exactly what would be broken if that one failed, without even needing to read the code inside it. That's a genuinely valuable habit; a good test name alone often tells a teammate exactly what went wrong, months from now, without them having to read a single line further.

A quick, honest question worth asking yourself as you write tests: "what's a case that could plausibly break this function, that I haven't checked yet?" That question — not just testing the obvious happy path — is what actually starts catching real bugs before they reach production.

Try It Yourself: Write a test for the apply_discount function you built back in Chapter 6 (recreate it in cleaning.py if needed). Write two tests: one checking a normal discount is applied correctly, and one checking that calling it with no discount at all returns the original price, unchanged.

Lesson 9.4 — Hands-On: Add Tests to Your Week 1 Work

Time to go back through this week and put a real safety net under the code you've already written. This is where testing stops being a new topic and starts being a habit.

Let's test the customer ID deduplication logic from Chapter 4. First, turn it into a proper function in cleaning.py, since testing works best on functions with a clear input and output — exactly what we practiced in Chapter 6:

python
def find_ids_in_all_systems(*id_lists):
    sets = [set(ids) for ids in id_lists]
    result = sets[0]
    for s in sets[1:]:
        result = result & s
    return result

That *id_lists is new — it lets the function accept any number of lists, not just a fixed two or three. Don't worry about the mechanics of it deeply; just know it means "however many lists you hand me, I'll work with all of them."

Now, the test:

python
from cleaning import find_ids_in_all_systems

def test_find_ids_in_all_systems_returns_shared_ids_only():
    crm_ids = [101, 102, 103, 104]
    erp_ids = [103, 104, 105, 106]
    billing_ids = [104, 107, 101]

    result = find_ids_in_all_systems(crm_ids, erp_ids, billing_ids)

    assert result == {104}

def test_find_ids_in_all_systems_returns_empty_set_when_nothing_shared():
    result = find_ids_in_all_systems([1, 2], [3, 4])
    assert result == set()
Output / Note

test_cleaning.py ...... [100%] 6 passed in 0.01s

Notice that second test — checking what happens when there's no overlap at all. That's a genuinely easy case to forget to check, and exactly the kind of edge case that quietly breaks things later if nobody thought to test for it up front.

Now let's do the same for the CSV profiler from Chapter 8. Move it into cleaning.py if it isn't there already, then write a test that builds a small, known sample file, runs the profiler on it, and checks the result:

python
import csv
from cleaning import profile_csv

def test_profile_csv_counts_missing_values(tmp_path):
    file_path = tmp_path / "test_orders.csv"
    with open(file_path, "w", newline="") as file:
        writer = csv.writer(file)
        writer.writerow(["order_id", "customer_name"])
        writer.writerow(["1", "Priya Shah"])
        writer.writerow(["2", ""])

    # profile_csv currently prints instead of returning —
    # that's a real design decision worth revisiting once you see this test

Pause here for a second, because this is a genuinely important, honest lesson, not a mistake to gloss over. Go back and look at profile_csv from Lesson 8.4 — it print()s its results, but never returns anything. And a test can only assert against something a function actually returns — it can't check what got printed to the screen.

This is a real, common turning point once you start testing seriously: it often reveals that a function needs a small redesign to be properly testable. Let's fix profile_csv so it returns its findings as a dictionary, and prints them separately:

python
def profile_csv(file_path):
    with open(file_path, "r") as file:
        reader = csv.DictReader(file)
        rows = list(reader)

    column_names = reader.fieldnames
    missing_counts = {}
    for column in column_names:
        missing_counts[column] = sum(1 for row in rows if row[column] == "")

    return {
        "row_count": len(rows),
        "columns": column_names,
        "missing_counts": missing_counts,
    }

Now the test can properly check it:

python
def test_profile_csv_counts_missing_values(tmp_path):
    file_path = tmp_path / "test_orders.csv"
    with open(file_path, "w", newline="") as file:
        writer = csv.writer(file)
        writer.writerow(["order_id", "customer_name"])
        writer.writerow(["1", "Priya Shah"])
        writer.writerow(["2", ""])

    result = profile_csv(str(file_path))

    assert result["row_count"] == 2
    assert result["missing_counts"]["customer_name"] == 1
Output / Note

1 passed in 0.02s

That tmp_path argument is something pytest provides automatically — a temporary, throwaway folder your test can safely write files into, without cluttering your real project or leaving test files lying around afterward. It's a small, genuinely handy tool worth recognizing.

Try It Yourself: Go back to your is_valid_email tests from Lesson 9.3, and add one more: a test proving that an email with extra spaces around it, like " priya.shah@company.com ", still correctly returns True — confirming the cleaning step inside the function is actually doing its job.