Python Foundation for Data Engineers

Chapter 12 — Meet Pandas: Series & DataFrames

Lesson 12.1 — What Pandas Is, and Why Data Engineers Live In It

If there's one tool that defines what it looks like to work with data in Python day to day, it's pandas. You've spent Week 1 building the raw skills — variables, loops, functions, files — and Chapter 11 showing you why fast, vectorized thinking beats loops. Pandas is where all of that comes together into the actual, real, everyday tool that data engineers open first thing every morning.

Here's the simplest way to think about it, and it's not an exaggeration: pandas is what happens when you take the list-of-dictionaries shape from Week 1, Chapter 4, and give it superpowers. Remember this, from back then?

python
customers = [
    {"id": 101, "name": "Priya Shah"},
    {"id": 102, "name": "Raj Kumar"},
]

That's genuinely, structurally, what most real data looks like — a batch of records, each one a set of named fields. Pandas takes exactly that shape, and gives it a proper name — a DataFrame — along with a huge, fast toolkit for filtering it, cleaning it, grouping it, and transforming it, all without writing loops by hand.

Let's install it and see that connection made concrete:

bash
pip install pandas
python
import pandas as pd

customers = [
    {"id": 101, "name": "Priya Shah"},
    {"id": 102, "name": "Raj Kumar"},
]

df = pd.DataFrame(customers)
print(df)
Output / Note

id name 0 101 Priya Shah 1 102 Raj Kumar

Same data you already knew how to build in Week 1. One new line, pd.DataFrame(customers), and it's transformed into a proper table — with row numbers on the left, automatically, and your dictionary keys turned into real column headers. That as pd nickname, same idea as np last chapter, is another universal convention you'll type at the top of nearly every script from here forward.

Over this week, you're going to learn to use a DataFrame to do everything you already know how to do by hand — filter rows, check for missing values, group and total things up, clean messy text — except faster, in fewer lines, and using tools purpose-built for exactly this kind of work.

Try It Yourself: Before the next lesson, look back at any list of dictionaries you wrote during Week 1 — the pipeline runs from Chapter 5, or the CSV rows from Chapter 8. Keep it in mind; you'll turn it into a real DataFrame very soon.

Lesson 12.2 — Series: A Single Labeled Column

Before we go further with full tables, let's meet the building block a DataFrame is actually made of: the Series. Think of a Series as one single column — a list of values, but with something extra a plain list never had: a label for every single position.

python
import pandas as pd

prices = pd.Series([24.99, 15.50, 89.99, 8.25])
print(prices)
Output / Note

0 24.99 1 15.50 2 89.99 3 8.25 dtype: float64

Notice those numbers on the left — 0, 1, 2, 3. That's called the index, and by default, pandas numbers it automatically, starting at zero, just like list indexing from Chapter 4. But unlike a plain list, you can give a Series a meaningful label instead:

python
prices = pd.Series(
    [24.99, 15.50, 89.99, 8.25],
    index=["mouse", "keyboard", "monitor", "cable"]
)
print(prices)
print(prices["monitor"])
Output / Note

mouse 24.99 keyboard 15.50 monitor 89.99 cable 8.25 dtype: float64 89.99

Now you can look a value up by a meaningful name, "monitor", instead of a bare position — genuinely similar to how dictionary keys worked back in Chapter 4, but with the fast, vectorized power from Chapter 11 built right in:

python
print(prices * 0.9)
print(prices[prices > 20])
Output / Note

mouse 22.491 keyboard 13.950 monitor 80.991 cable 7.425 dtype: float64 mouse 24.99 monitor 89.99 dtype: float64

Recognize that second line? It's the exact boolean mask pattern from Chapter 11, working exactly the same way here, on a labeled Series. Everything you learned about NumPy arrays carries forward directly — a Series is genuinely built on top of a NumPy array underneath, with labels added on top.

You won't spend most of your time working with a lone Series — in real work, you're almost always dealing with many columns together. But every single column inside a DataFrame, which we're about to properly meet, actually is a Series. Understanding this piece first is what makes the whole picture click.

Try It Yourself: Create a Series called row_counts with values [4500, 0, 3200, 980], indexed by ["CRM", "ERP", "Billing", "Marketing"]. Print the value for "Billing" by name, and print only the systems with a row count greater than 0.

Lesson 12.3 — DataFrame: Rows and Columns Together

Now let's properly meet the DataFrame — a full table, made up of multiple Series, all sharing the same row labels, sitting side by side as columns. This is the object you'll spend the vast majority of this entire course working with.

python
import pandas as pd

data = {
    "product": ["Mouse", "Keyboard", "Monitor", "Cable"],
    "price": [24.99, 15.50, 89.99, 8.25],
    "quantity": [3, 1, 5, 2],
}

df = pd.DataFrame(data)
print(df)
Output / Note

product price quantity 0 Mouse 24.99 3 1 Keyboard 15.50 1 2 Monitor 89.99 5 3 Cable 8.25 2

Notice the shape of data here — a dictionary where each key is a column name, and each value is a list of that column's values, top to bottom. This is the second common way to build a DataFrame, alongside the list-of-dictionaries shape from Lesson 12.1 — both are genuinely common in real work, so it's worth recognizing both on sight.

A few things worth knowing immediately, because you'll use them constantly, in nearly every script from here forward:

python
print(df.shape)
print(df.columns)
print(df.dtypes)
Output / Note

(4, 3) Index(['product', 'price', 'quantity'], dtype='object') product object price float64 quantity int64 dtype: object

.shape gives you rows and columns as a pair — (4, 3) means 4 rows, 3 columns, a fast sanity check you'll run on nearly every new file you load. .columns lists the column names. .dtypes shows the type of each column — notice price correctly came in as float64 and quantity as int64, because we built this DataFrame from real Python numbers, not text pulled from a file. You'll see this look very different once we start reading messy CSV files properly next chapter.

Grabbing a single column pulls out exactly the Series you met last lesson:

python
print(df["price"])
print(type(df["price"]))
Output / Note

0 24.99 1 15.50 2 89.99 3 8.25 Name: price, dtype: float64 <class 'pandas.core.series.Series'>

That confirms it directly: a DataFrame column really is a Series, with all the same tools you just practiced — including vectorized math and boolean masks:

python
df["price_with_tax"] = df["price"] * 1.08
print(df)
Output / Note

product price quantity price_with_tax 0 Mouse 24.99 3 26.9892 1 Keyboard 15.50 1 16.7400 2 Monitor 89.99 5 97.1892 3 Cable 8.25 2 8.9100

That last line just added a brand new column to the table, calculated from an existing one, in a single line — no loop, no manually building a new list and reattaching it. This is genuinely how real, everyday pandas work looks: read data in, calculate new columns directly from existing ones, and keep building up the table.

Try It Yourself: Build a DataFrame from this dictionary: {"system": ["CRM", "ERP", "Billing"], "rows_loaded": [4500, 3200, 0]}. Print its .shape and .dtypes. Then add a new column, has_data, that's True where rows_loaded is greater than 0, and False otherwise — using a boolean comparison, the same way you built price_with_tax above.

Lesson 12.4 — Hands-On: Build a DataFrame from Real-Looking Records

Let's close this chapter by making the connection to Week 1 completely explicit — taking data in the exact shape you built by hand back then, and turning it into a proper, working DataFrame.

Here's a batch of pipeline run records — genuinely the same shape as the pipeline health data from Week 1, Chapter 5:

python
pipeline_runs = [
    {"name": "orders_sync", "status": "success", "rows": 4500},
    {"name": "customer_sync", "status": "failed", "rows": 0},
    {"name": "inventory_sync", "status": "success", "rows": 12},
    {"name": "billing_sync", "status": "success", "rows": 3200},
]

Back in Chapter 5, you classified these with a for loop and a chain of if/elif/else. Let's do the same job with a DataFrame instead, and see how the shape of the solution changes.

python
import pandas as pd

df = pd.DataFrame(pipeline_runs)
print(df)
Output / Note
         name   status  rows

0 orders_sync success 4500 1 customer_sync failed 0 2 inventory_sync success 12 3 billing_sync success 3200

Now, let's add a health column, using the same rules from Chapter 5 — but as a vectorized calculation instead of a loop. We'll use np.select, a NumPy tool that's genuinely useful for exactly this kind of "several conditions, several results" situation:

python
import numpy as np

conditions = [
    df["status"] == "failed",
    df["rows"] < 100,
]
choices = ["Failed", "Warning"]

df["health"] = np.select(conditions, choices, default="Healthy")
print(df)
Output / Note
         name   status  rows   health

0 orders_sync success 4500 Healthy 1 customer_sync failed 0 Failed 2 inventory_sync success 12 Warning 3 billing_sync success 3200 Healthy

Read np.select like a sentence: "check each condition in order, and use the matching choice — if failed, label it Failed; else if rows is under 100, label it Warning; otherwise, use the default, Healthy." Compare this directly to your Chapter 5 solution — same logic, same result, but every row was classified at once, instead of one at a time through a loop.

This is genuinely the shift this whole week is about: not new logic, but a new, faster, more direct way to express logic you already understand deeply, because you built it by hand first.

Let's finish with a quick, honest summary — something you'll do constantly once real files are involved:

python
print(df["health"].value_counts())
Output / Note

Healthy 2 Warning 1 Failed 1 Name: health, dtype: int64

.value_counts() counts how many times each unique value appears in a column — a single line that just replaced the manual counting loop from Chapter 5's "Try It Yourself" exercise. You'll reach for this constantly, on almost any categorical column you meet from here on.

Try It Yourself: Add a fifth pipeline run to pipeline_runs, with "status": "success" and "rows": 0 — the exact dangerous edge case from Week 1, Chapter 5's closing exercise. Rebuild the DataFrame, and adjust the conditions/choices in np.select so this row is also correctly caught as a "Warning", exactly as you did by hand back then.