Chapter 17 — Validating & Testing DataFrames

Lesson 17.1 — Why "It Looks Right" Still Isn't Enough for a DataFrame

Back in Week 1, Chapter 9, you learned the core lesson of testing: looking at output once, and deciding it "looks right," tells you nothing about tomorrow's file. That lesson matters even more now, with everything you've built this week, because a DataFrame is so much bigger than a single value — and it's genuinely easy to eyeball ten rows, see they look fine, and miss something wrong buried in row four thousand.

Think about everything you've built since Chapter 14: cleaning functions that strip and fix names, convert types, parse dates, handle missing values. Every one of those functions makes decisions — what counts as a duplicate, what a missing value should become, which date format to trust. If any of those decisions is subtly wrong, print(df.head()) will almost never catch it. .head() only shows you the first five rows, and real bugs love to hide in the rows you didn't happen to look at.

This chapter takes the exact testing habit from Week 1 and extends it to DataFrames properly — checking not just "does this function run without crashing," but "does it produce exactly the table I actually expect, every time, automatically."

Here's the same honest question from Week 1, Chapter 9.3, worth asking again now, on a bigger scale: what's a case that could plausibly break one of your cleaning functions, that a quick glance at .head() would never reveal? That question is exactly what the rest of this chapter is built to help you answer, systematically, instead of by luck.

Lesson 17.2 — A Light Touch on Schema Validation: Pydantic & Pandera

Before we get to testing your own functions, it's worth knowing about a related idea: checking that a DataFrame's shape is correct — right columns, right types, values within reasonable bounds — before you even start working with it. This is called schema validation, and two tools come up constantly in real data engineering work. We'll keep this lesson deliberately brief — recognizing what these tools are for is the goal here, not deep mastery.

Pydantic validates a single structured record — genuinely useful for checking one API response, one config file, one incoming row, against a definition of what it should look like:


python
from pydantic import BaseModel

class Order(BaseModel):
    order_id: int
    customer: str
    total: float

order = Order(order_id=10432, customer="Priya Shah", total=149.50)
print(order)

Output / Note

order_id=10432 customer='Priya Shah' total=149.5

Try handing it something that doesn't fit, and it tells you exactly what's wrong, immediately — the same honest, specific-error philosophy from Week 1, Chapter 7's raise lesson:


python
try:
    bad_order = Order(order_id="not-a-number", customer="Priya Shah", total=149.50)
except Exception as error:
    print(error)

Output / Note

1 validation error for Order order_id Input should be a valid integer

Pandera does the equivalent job, but for an entire DataFrame at once — checking every row against a schema in one pass, rather than one record at a time:


python
import pandas as pd
import pandera.pandas as pa

schema = pa.DataFrameSchema({
    "order_id": pa.Column(int),
    "customer": pa.Column(str),
    "total": pa.Column(float, pa.Check.greater_than(0)),
})

df = pd.DataFrame({
    "order_id": [10432, 10433],
    "customer": ["Priya Shah", "Raj Kumar"],
    "total": [149.50, 89.99],
})

validated = schema.validate(df)
print("Schema passed")

Output / Note

Schema passed

Notice pa.Check.greater_than(0) on the total column — that's checking not just the type, but a real business rule: a total should never be zero or negative. Hand it data that breaks that rule, and it tells you exactly which rows failed, and why — rather than letting a bad value quietly slip through into a report.

Here's the honest distinction worth remembering: Pydantic is for validating one record at a time, often at the edge of your pipeline — an incoming API call, a single config. Pandera is for validating a whole DataFrame at once, often right after a cleaning step, to confirm the output actually meets the rules you expect before it moves further downstream. You won't need to master either deeply for this course, but recognizing them, and the specific gap each one fills, will serve you well the moment you meet a real pipeline that uses them.

Try It Yourself: Using the Pandera example above, add a second check to the order_id column requiring it to be greater than 0. Then deliberately create a DataFrame with an order_id of -5, and run it through schema.validate() to see the specific error message pandera gives you.

Lesson 17.3 — Testing a Cleaning Function with assert_frame_equal

Now let's properly test the kind of function you've been writing all chapter — one that takes a messy DataFrame and returns a clean one. The core pattern is exactly the one from Week 1, Chapter 9.3 — known input in, expected answer out — just applied to a whole table instead of a single value.

First, let's set up a small, testable cleaning function, the same shape as Chapter 14's work:


python
import pandas as pd

def clean_names(df):
    df = df.copy()
    df["customer"] = df["customer"].str.strip().str.title()
    return df

Notice df.copy() right at the top — that's a genuinely important habit worth building now. Without it, this function would modify the original DataFrame the caller handed in, which can cause confusing, hard-to-trace bugs elsewhere in a larger pipeline. Copying first means this function only changes the version it hands back.

Now, the test. pandas provides a purpose-built tool for comparing two DataFrames properly: pd.testing.assert_frame_equal().


python
import pandas as pd
from cleaning import clean_names

def test_clean_names_strips_and_titles():
    input_df = pd.DataFrame({"customer": ["  priya shah  ", "RAJ KUMAR"]})
    expected_df = pd.DataFrame({"customer": ["Priya Shah", "Raj Kumar"]})

    result_df = clean_names(input_df)

    pd.testing.assert_frame_equal(result_df, expected_df)

Output / Note

1 passed in 0.02s

Read it exactly the way you read a plain assert back in Week 1 — build a small, known input, decide exactly what the correct output should look like, run your function, and check the two match. The difference is that assert result == expected doesn't work reliably on DataFrames — comparing two tables properly means checking every value, every column, every row, and assert_frame_equal does exactly that, correctly, in one line.

A genuinely useful detail: if the test fails, assert_frame_equal gives you a specific, detailed report of exactly which values differed — not just "these don't match," but precisely where and how, which saves real debugging time. Let's see it on purpose:


python
def test_clean_names_wrong_expectation():
    input_df = pd.DataFrame({"customer": ["  priya shah  "]})
    expected_df = pd.DataFrame({"customer": ["priya shah"]})

    result_df = clean_names(input_df)

    pd.testing.assert_frame_equal(result_df, expected_df)

Output / Note

AssertionError: DataFrame.iloc[:, 0] (column name="customer") are different

DataFrame.iloc[:, 0] (column name="customer") values are different (100.0 %) [left]: [Priya Shah] [right]: [priya shah]

That message tells you precisely which column, which values, and exactly what was expected versus what was actually produced — genuinely more useful than staring at two printed tables side by side, trying to spot the difference yourself.

Try It Yourself: Write a function remove_duplicate_orders(df) that calls .drop_duplicates() on a DataFrame. Write a test for it using assert_frame_equal, with a small input DataFrame containing one obvious duplicate row, and an expected output with that duplicate removed.

Lesson 17.4 — Hands-On: Add Tests to Chapter 14's Cleaning Function

Let's go back to the signature exercise from Chapter 14 — cleaning that messy multi-source sales dataset — and put a proper test suite underneath it, exactly the way Week 1, Lesson 9.4 put tests under your Week 1 work.

First, turn Chapter 14's cleaning steps into one proper, testable function in cleaning.py:


python
import pandas as pd

def clean_sales(df):
    df = df.copy()

    df = df.drop_duplicates()

    df["customer"] = df["customer"].str.strip().str.title()
    df["customer"] = df["customer"].fillna("Unknown Customer")

    df["total"] = pd.to_numeric(df["total"], errors="coerce")
    df["order_date"] = pd.to_datetime(df["order_date"])

    return df

Notice this is genuinely just Chapter 14's cleaning steps, lifted directly into a proper function — nothing new here except giving it a name and a clear input/output shape, exactly the same move you made with is_valid_email back in Week 1, Chapter 6.

Now, the tests. Let's check a few different things this function needs to get right, each with its own small, focused test:


python
import pandas as pd
from cleaning import clean_sales

def test_clean_sales_removes_exact_duplicates():
    input_df = pd.DataFrame({
        "order_id": [1, 1],
        "customer": ["Priya Shah", "Priya Shah"],
        "total": ["149.50", "149.50"],
        "order_date": ["2026-01-14", "2026-01-14"],
    })

    result = clean_sales(input_df)

    assert len(result) == 1

def test_clean_sales_cleans_customer_names():
    input_df = pd.DataFrame({
        "order_id": [1],
        "customer": ["  priya shah  "],
        "total": ["149.50"],
        "order_date": ["2026-01-14"],
    })

    result = clean_sales(input_df)

    assert result["customer"].iloc[0] == "Priya Shah"

def test_clean_sales_converts_bad_total_to_nan():
    input_df = pd.DataFrame({
        "order_id": [1],
        "customer": ["Priya Shah"],
        "total": ["unknown"],
        "order_date": ["2026-01-14"],
    })

    result = clean_sales(input_df)

    assert pd.isna(result["total"].iloc[0])

def test_clean_sales_parses_multiple_date_formats():
    input_df = pd.DataFrame({
        "order_id": [1, 2],
        "customer": ["Priya Shah", "Raj Kumar"],
        "total": ["149.50", "89.99"],
        "order_date": ["2026-01-14", "2026/01/15"],
    })

    result = clean_sales(input_df)

    assert result["order_date"].dtype == "datetime64[ns]"

Output / Note

test_cleaning.py .... [100%] 4 passed in 0.03s

Notice each test checks exactly one behavior, with a small, focused input built specifically to trigger it — the same discipline from Week 1, Chapter 9.3. test_clean_sales_converts_bad_total_to_nan doesn't try to test everything about the function at once — it isolates the one thing it's checking, a bad total, and confirms exactly the expected, honest outcome: not a crash, not a silently wrong number, but a properly visible NaN.

This is genuinely what real, professional testing of pandas code looks like: several small, sharply focused tests, each one proving a specific promise the function makes, rather than one giant test trying to check everything simultaneously and telling you little when it fails.

Try It Yourself: Write one more test, test_clean_sales_fills_missing_customer_name, using an input row where customer is None, confirming the result correctly becomes "Unknown Customer". Run the full test file and confirm all five tests pass.