Chapter 4 — Collections: Storing Groups of Data

Lesson 4.1 — Lists: Your Everyday Container

So far, every variable you've written has held exactly one thing — one name, one price, one row count. But real data almost never shows up one piece at a time. It shows up as a batch — a whole file of customer names, a whole column of order totals, a whole day's worth of log lines. You need somewhere to put all of it at once.

That's what a list is for. Think of it as a single container that holds many values, in order, under one name.

pythonCopy
source_systems = ["CRM", "ERP", "Billing", "Support"]

print(source_systems)
print(len(source_systems))

Output / Note

['CRM', 'ERP', 'Billing', 'Support'] 4

Notice the square brackets, and the commas separating each item. len() — the same tool you used on strings back in Chapter 3 — works here too, and tells you how many items are in the list. You'll use len() on lists constantly, for things like "how many rows did we actually load?"

Getting one item out of a list uses its position, called an index, in square brackets:

pythonCopy
print(source_systems[0])
print(source_systems[1])
print(source_systems[-1])

Output / Note

CRM ERP Support

Here's the detail that trips up nearly every beginner, so let's get it right from day one: Python counts starting at zero, not one. So source_systems[0] is the first item, [1] is the second, and so on. And [-1] is a shortcut for "the last item," which is genuinely useful — you'll reach for it often, instead of counting all the way to the end yourself.

Lists aren't fixed once you make them. You can add to them:

pythonCopy
source_systems.append("Marketing")
print(source_systems)

Output / Note

['CRM', 'ERP', 'Billing', 'Support', 'Marketing']

.append() adds one new item to the end. And you can check whether something is already in a list:

pythonCopy
print("CRM" in source_systems)
print("Finance" in source_systems)

Output / Note

True False

That in check is something you'll use all the time — checking whether a value has already shown up before, whether a file has already been processed, whether an ID is one you've seen. Keep it in your back pocket; it comes up again very soon.

Try It Yourself: Create a list called pending_files with three made-up CSV file names in it. Print how many files are in the list. Then .append() one more file, check whether a specific file name is in the list using in, and print the last file in the list using [-1].

---

Lesson 4.2 — Tuples & Sets: When Order or Uniqueness Matters

Lists are your default, everyday container. But there are two close relatives worth knowing, because each one solves a specific problem lists don't handle as well.

First, tuples. A tuple looks almost identical to a list, but with round brackets instead of square ones — and once you create it, it can't be changed.

pythonCopy
db_config = ("localhost", 5432, "orders_db")
print(db_config)
print(db_config[1])

Output / Note

('localhost', 5432, 'orders_db') 5432

Why would you want something you can't change? Think about a database connection — a host, a port, a database name. Once that's set for a script, it shouldn't accidentally get modified halfway through running. A tuple locks that in, on purpose, as a small safety net. You won't use tuples nearly as often as lists in this course, but you'll recognize them — plenty of Python tools, including ones you'll meet later, hand you back tuples without asking.

Now, sets. This one's genuinely useful, and it solves a problem you'll run into constantly in data work: getting rid of duplicates.

Picture this: three different source systems each send you a list of customer IDs, and plenty of the same customers show up in more than one file.

pythonCopy
crm_ids = [101, 102, 103, 104]
erp_ids = [103, 104, 105, 106]

unique_ids = set(crm_ids) | set(erp_ids)
print(unique_ids)
print(len(unique_ids))

Output / Note

{104, 105, 101, 102, 103, 106} 6

A set is a collection that automatically throws away duplicates and doesn't care about order — think of it as a bouncer that simply refuses to let the same ID in twice. The | symbol combines two sets together, keeping only one copy of anything that appears in both. Notice, too, that the printed order looks scrambled compared to how you typed them in — that's normal. Sets don't preserve order, because order isn't the point; uniqueness is.

You'll also see the exact number of matching duplicates between two sources, using & instead of |:

pythonCopy
shared_ids = set(crm_ids) & set(erp_ids)
print(shared_ids)

Output / Note

{103, 104}

That single line just told you exactly which customers exist in both the CRM and the ERP system — a genuinely common, real question in data engineering, answered in one line instead of a pile of manual comparisons.

Try It Yourself: Create two lists of made-up product IDs, with at least two IDs overlapping between them. Use set() and | to find the full combined list of unique product IDs, and & to find just the ones that appear in both lists.

---

Lesson 4.3 — Dictionaries: Fast Lookups with Keys

Here's a limitation of lists worth noticing: to get a value out, you need to know its position — its index. But real data usually isn't about position, it's about a name. You don't think of a customer's email as "the third thing in the list." You think of it as "their email."

That's exactly the gap a dictionary fills. Instead of a position, every value gets a name, called a key.

pythonCopy
customer = {
    "id": 10432,
    "name": "Priya Shah",
    "email": "priya.shah@company.com",
    "is_priority": True
}

print(customer["name"])
print(customer["email"])

Output / Note

Priya Shah priya.shah@company.com

You saw this exact shape back in Chapter 2's hands-on exercise, with the curly braces — now you know its proper name, and how it really works. Each key: value pair is one entry. You look things up by key, not position, which reads far more naturally once your data actually has a name for each piece.

You can check whether a key exists before trying to use it, which matters a lot with real, messy data where a field might simply be missing:

pythonCopy
print("phone" in customer)

Output / Note

False

Trying to grab a key that doesn't exist crashes your program with a KeyError. A much safer habit, one working engineers use constantly, is .get(), which lets you provide a fallback instead:

pythonCopy
phone = customer.get("phone", "Not provided")
print(phone)

Output / Note

Not provided

That one line just saved your script from crashing on a perfectly normal, common situation — a field that simply wasn't filled in. You'll lean on .get() heavily once we start reading real, messy files later in this course.

Dictionaries also show up constantly nested inside lists — this is genuinely what a lot of real-world data looks like, a whole batch of records, each one a dictionary:

pythonCopy
customers = [
    {"id": 101, "name": "Priya Shah"},
    {"id": 102, "name": "Raj Kumar"},
]

for c in customers:
    print(c["name"])

Output / Note

Priya Shah Raj Kumar

Don't worry about that for line yet — we're covering loops properly next chapter. For now, just notice the shape: a list of dictionaries is exactly how a batch of records tends to look once it's loaded into Python, and you'll see this pattern again and again from here on.

Try It Yourself: Create a dictionary called order with keys order_id, customer_name, and total. Print the customer_name using its key. Then use .get() to look up a "discount_code" key that doesn't exist, with a fallback value of "None applied".

---

Lesson 4.4 — Comprehensions: The Pythonic Shortcut

Here's a pattern you'll write over and over in data work: take a list, do something to every item in it, and collect the results into a new list. There's a shorter, cleaner way to write that pattern than you might expect, and Python programmers reach for it constantly — so much so that not knowing it makes your code visibly stand out as unfinished.

Let's start with the long way, using a for loop, just so you can see exactly what's being shortened — again, don't worry about fully understanding for loops yet, just follow the shape:

pythonCopy
names = ["priya shah", "raj kumar", "amit verma"]

cleaned = []
for name in names:
    cleaned.append(name.title())

print(cleaned)

Output / Note

['Priya Shah', 'Raj Kumar', 'Amit Verma']

Four lines, to do one simple thing: clean up every name in the list. Here's the same result, written as a list comprehension:

pythonCopy
names = ["priya shah", "raj kumar", "amit verma"]
cleaned = [name.title() for name in names]
print(cleaned)

Output / Note

['Priya Shah', 'Raj Kumar', 'Amit Verma']

Read it left to right, like a sentence: "give me name.title(), for every name in names." One line, same result. This isn't just about typing less — code like this is genuinely easier to read at a glance once it becomes familiar, because the whole operation sits in one place instead of being spread across a loop.

You can add a condition too, filtering as you go:

pythonCopy
row_counts = [4500, 0, 3200, 0, 980]
non_empty = [count for count in row_counts if count > 0]
print(non_empty)

Output / Note

[4500, 3200, 980]

That single line just filtered out every empty file from a batch — a genuinely common first step before processing a day's worth of files.

The same idea works for dictionaries too, building one from a list:

pythonCopy
names = ["CRM", "ERP", "Billing"]
name_lengths = {name: len(name) for name in names}
print(name_lengths)

Output / Note

{'CRM': 3, 'ERP': 3, 'Billing': 7}

A quick word of honest advice: comprehensions are wonderful for simple, one-step operations. Once the logic inside gets complicated, a regular for loop is usually clearer — and clearer code is always worth more than clever code. You'll get a feel for that balance with practice.

Try It Yourself: Given prices = [24.99, 0, 15.50, 0, 8.25], write a list comprehension that keeps only the prices greater than 0. Then write a second comprehension that converts a list of source system names to all lowercase.

---

Lesson 4.5 — Hands-On: Deduplicate Customer IDs

Let's bring this whole chapter together on a task that comes up all the time in real data engineering work: three source systems, each sending you their own list of customer IDs, with plenty of overlap between them.

pythonCopy
crm_ids = [101, 102, 103, 104, 102]
erp_ids = [103, 104, 105, 106]
billing_ids = [104, 107, 101]

Notice 102 shows up twice in the CRM list alone — a duplicate within a single source, not just across sources. Real exports do this more often than you'd think, usually from a system re-sending the same record after an update.

Your task:

1. Combine all three lists into one master list of every ID that appears anywhere.
2. Turn that into a set, to get the unique customer IDs across all systems.
3. Figure out which customers appear in all three systems — the most reliably confirmed records.
4. Print a short, clear summary.

Here's the shape to build from:

pythonCopy
crm_ids = [101, 102, 103, 104, 102]
erp_ids = [103, 104, 105, 106]
billing_ids = [104, 107, 101]

all_ids = set(crm_ids) | set(erp_ids) | set(billing_ids)
in_all_three = set(crm_ids) & set(erp_ids) & set(billing_ids)

print(f"Total unique customers across all systems: {len(all_ids)}")
print(f"Customers confirmed in all three systems: {in_all_three}")

Output / Note

Total unique customers across all systems: 7 Customers confirmed in all three systems: {104}

Seven unique customers total, but only customer 104 shows up in every single system. In a real job, that second number matters a lot — it's often the group you can trust most, because three independent systems all agree they exist.

One more useful step: which customers appear in only one system, and might be worth double-checking?

pythonCopy
only_crm = set(crm_ids) - set(erp_ids) - set(billing_ids)
print(f"Customers only in CRM: {only_crm}")

Output / Note

Customers only in CRM: {102}

The - symbol subtracts one set from another — "everything in the first set, except anything that also shows up in the others." You've now used four different set operations — combine, find shared, find exclusive, subtract — to turn three messy lists into real, trustworthy answers. That's not a toy exercise. That's exactly the kind of reconciliation question a data engineer gets asked in their very first week on the job.

Try It Yourself: Add a fourth source system, support_ids, with a small made-up list of your own, including at least one ID that overlaps with crm_ids. Update the code to find which customers now appear in all four systems, and print how many unique customers exist across all four combined.