Python Foundation for Data Engineers

Chapter 3 — Working with Text

Lesson 3.1 — String Basics

Here's something worth knowing before we go any further: most of what a data engineer actually cleans up, day to day, is text. Not fancy math. Not complicated algorithms. Just messy, inconsistent text that needs to be tidied up before anyone can trust it.

Picture a customer names column from a signup form. One person typed their name in all lowercase. Another left a space at the start by accident. Someone else typed "N/A" instead of leaving it blank. None of this is unusual — it's just what real data looks like before someone cleans it. That someone is about to be you.

In Python, any piece of text is called a string. You've already seen strings — anything wrapped in quotes, like "Priya Shah" or "CRM". Let's learn how to actually work with them.

Strings have built-in methods — little tools attached to every string, ready to use with a dot:

python
name = "  priya shah  "

print(name.strip())
print(name.strip().title())
print(name.upper())
print(name.lower())
Output / Note

priya shah Priya Shah PRIYA SHAH
priya shah

Notice something important here: .strip() removes extra spaces from the start and end. .title() capitalizes each word properly. .upper() and .lower() change the case entirely. And notice that upper() and lower(), on their own, didn't remove those extra spaces — each method only does the one job it's named for. This is exactly why we chained .strip() and .title() together in that second line — clean first, then format.

A few more you'll reach for constantly:

python
raw = "Order-ID: 10432"

print(raw.replace("Order-ID: ", ""))
print(raw.split(": "))
print(len(raw))
Output / Note

10432 ['Order-ID', '10432'] 16

.replace() swaps one piece of text for another — perfect for stripping out a label you don't need. .split() breaks a string apart wherever it finds the piece you tell it, and hands you back a list of the pieces — we'll get properly into lists next chapter, but you'll already recognize this shape when we get there. And len() tells you how many characters are in the string, which turns out to be surprisingly useful for spotting bad data, like a phone number that's three digits short.

One more thing worth knowing now, because it trips people up later: strings in Python don't change in place. .strip() doesn't clean the original string — it hands you back a brand new, cleaned one. If you want to keep the clean version, you have to store it:

python
name = "  priya shah  "
name = name.strip().title()
print(name)
Output / Note

Priya Shah

See that we stored the result back into name? Without that, the original messy version would still be sitting there, untouched, and you'd wonder why your "cleaning" didn't seem to work.

Try It Yourself: Create a variable raw_city = " new york ". Clean it so it reads as "New York" — no extra spaces, properly capitalized. Then create product_code = "SKU_10432_WIDGET" and use .split("_") to break it into its three pieces, and print each one.

Lesson 3.2 — f-strings: Speaking Python's Language

Every data engineer writes messages constantly — status updates, log lines, error messages, little summaries printed while a script runs. You've already glued text together with +, but it gets clumsy fast, especially once numbers are involved. There's a much better way, and once you learn it, you'll use it in nearly every script you write from here on.

It's called an f-string — short for "formatted string." Here's the idea:

python
source_system = "CRM"
row_count = 4500

message = f"Loaded {row_count} rows from {source_system}"
print(message)
Output / Note

Loaded 4500 rows from CRM

See that little f right before the opening quote? That's what tells Python: "inside this string, anything wrapped in curly braces isn't plain text — go fetch that variable's value and drop it in here." No more stitching pieces together with +, no more converting numbers to text by hand with str(). Python handles all of that quietly, for you, inside the braces.

You can even do quick calculations right inside the braces:

python
total_rows = 4500
bad_rows = 12

print(f"{bad_rows} bad rows out of {total_rows} ({bad_rows / total_rows:.2%} error rate)")
Output / Note

12 bad rows out of 4500 (0.27% error rate)

That :.2% bit is a formatting instruction — it tells Python "treat this number as a percentage, and show two decimal places." You don't need to memorize every formatting code today. Just know they exist, and that you can look one up whenever you need it. That's a completely normal, everyday part of how working engineers write code — nobody has it all memorized.

f-strings become especially useful once we start building real pipeline scripts later in this course, where you'll want to print out things like "Processing file: orders_2026_01.csv" or "Warning: 3 rows skipped due to missing price" — clear, specific messages that tell you exactly what your code is doing while it runs.

Try It Yourself: Create variables file_name = "orders_jan.csv", rows_loaded = 812, and rows_skipped = 5. Using one f-string, print a single sentence that mentions all three — something like: "orders_jan.csv: 812 rows loaded, 5 skipped."

Lesson 3.3 — A Gentle Intro to Regex

Let's talk about a tool that looks intimidating the first time you see it, but is genuinely one of the most useful things you'll learn in this entire course: regex, short for regular expressions.

Here's the problem regex solves. Say you've got a column of values, and you need to find which ones look like a valid email address. You could check character by character with a pile of if statements — but that gets messy and fragile fast. Regex lets you describe a pattern — "text, then an @ symbol, then more text, then a dot, then a few letters" — and let Python search for anything matching that shape.

We'll keep this lesson deliberately light. Regex is a deep topic, and full mastery isn't the goal today — recognizing the pattern and knowing where to reach for it is.

Python's regex tools live in a module called re. Let's check if a string looks like a valid email:

python
import re

email = "priya.shah@company.com"

if re.search(r"[\w.]+@[\w.]+\.\w+", email):
    print("Looks like a valid email")
else:
    print("Not a valid email")
Output / Note

Looks like a valid email

Let's slow down and read that pattern, piece by piece, because it looks like noise the first time you see it:

  • [\w.]+ means "one or more letters, numbers, underscores, or dots" — this matches the part before the @ symbol
  • @ matches an actual @ symbol
  • [\w.]+ again, for the part after the @ — the company name
  • \. matches an actual dot — the backslash is needed because a plain dot means something else in regex
  • \w+ matches the letters at the end, like com or org

You don't need to write patterns like this from memory yet. What matters today is that you can recognize the shape of one, and know that re.search() is how you ask Python "does this pattern show up anywhere in this text?"

One more regex tool worth knowing now — pulling a piece out of a bigger string, not just checking if it matches:

python
import re

log_line = "2026-01-14 ERROR: file not found"

match = re.search(r"\d{4}-\d{2}-\d{2}", log_line)
print(match.group())
Output / Note

2026-01-14

Here, \d means "any digit," and {4} means "exactly four of those in a row." So \d{4}-\d{2}-\d{2} describes the shape of a date like 2026-01-14. This exact pattern — pulling a date or an ID out of a messy log line or a poorly formatted column — is something you'll do often in real pipeline work.

Try It Yourself: Given text = "Contact us at support@example.com or call 555-0148", use re.search() with the email pattern above to check whether an email address exists in the string, and print the result using .group().

Lesson 3.4 — Hands-On: Clean a Messy Contact List

Time to put this whole chapter to work on something that looks like a real task, because it is one. Cleaning up a badly entered contact list is one of the most common first assignments a junior data engineer gets handed.

Here are three raw entries, exactly as messy as real ones tend to be:

python
contacts = [
    "  PRIYA shah , priya.shah@company.com ",
    "raj KUMAR,raj_kumar@company.com  ",
    " Amit Verma , amit.verma@@company.com"
]

Notice the problems hiding in here: inconsistent spacing, inconsistent capitalization, and that third email has two @ symbols by mistake — a typo that would silently break something downstream if nobody caught it.

Your task, using what you've learned this chapter, for each contact:

  1. Split it into a name part and an email part, using .split(",").
  2. Clean the name with .strip() and .title().
  3. Clean the email with .strip() and .lower().
  4. Use regex to check whether the cleaned email actually looks valid.
  5. Print a clean, readable summary line for each contact.

Here's the shape to get you started:

python
import re

for contact in contacts:
    name_part, email_part = contact.split(",")
    name = name_part.strip().title()
    email = email_part.strip().lower()

    is_valid = bool(re.search(r"[\w.]+@[\w.]+\.\w+", email)) and email.count("@") == 1

    print(f"{name} — {email} — valid: {is_valid}")
Output / Note

Priya Shah — priya.shah@company.com — valid: True Raj Kumar — raj_kumar@company.com — valid: True Amit Verma — amit.verma@@company.com — valid: False

Look closely at that email.count("@") == 1 check we added. The regex pattern alone would still find a match inside amit.verma@@company.com, because there's technically a valid-looking piece hiding in there — so we added a second, simple check on top of it. This is a real habit of working data engineers: one clever check is rarely enough on its own. You layer a few simple, honest checks together, and trust the result more because of it.

You just built something that looks a lot like the first stage of a real data-cleaning pipeline — take messy input, clean each field, validate it, and produce a trustworthy result. That's not a toy exercise. That's the actual shape of the job.

Try It Yourself: Add a fourth contact to the list with a name in mixed-up spacing and case, and an email missing the @ symbol entirely, like "john doe john.doe.company.com". Run your cleaning loop again and confirm it correctly marks that one as invalid.