Python Foundation for Data Engineers

Chapter 8 — Reading & Writing Files

Lesson 8.1 — Working with Plain Text Files

Everything you've written so far has lived and died inside one running script. The moment your program ends, all of it — every variable, every list, every dictionary — disappears. Real data engineering work can't work that way. Files arrive from other systems, and your results need to be saved somewhere other systems can pick them up. That connection between your code and the outside world starts with reading and writing files.

Let's start simple, with a plain text file.

python
with open("log.txt", "w") as file:
    file.write("Pipeline started\n")
    file.write("Loaded 4500 rows\n")
    file.write("Pipeline finished\n")

Run this, and look in your python-de-foundations folder in VS Code — a new file called log.txt has appeared, with those three lines inside it.

Let's unpack that with open(...) as file: line, because you'll type some version of it constantly from here on. open("log.txt", "w") opens a file named log.txt in write mode — that's what the "w" means. If the file doesn't exist yet, Python creates it. If it does exist, this mode overwrites it completely, so use "w" carefully. The with part is important too: it guarantees the file gets properly closed and saved once you're done with it, even if something goes wrong partway through. You'll almost always open files this way, rather than manually opening and closing them yourself.

Notice that \n at the end of each line — that's a newline character, telling the file "start a new line here." Leave it out, and all three lines would run together into one unreadable mess.

Now let's read that file back:

python
with open("log.txt", "r") as file:
    contents = file.read()

print(contents)
Output / Note

Pipeline started Loaded 4500 rows Pipeline finished

"r" means read mode. .read() pulls in the entire file as one single string. Often, though, you want to work through a file one line at a time, which turns out to be even more common in real pipeline work:

python
with open("log.txt", "r") as file:
    for line in file:
        print("LOG:", line.strip())
Output / Note

LOG: Pipeline started LOG: Loaded 4500 rows LOG: Pipeline finished

Notice we looped straight over file itself — Python lets you walk through a file line by line, exactly like looping over a list. And notice the .strip() from Chapter 3 making a comeback — every line read from a file carries its own trailing \n, and stripping it off is a habit you'll repeat constantly.

One more mode worth knowing: "a", for append. It adds new lines to the end of a file without erasing what's already there — genuinely useful for something like a log file you want to keep adding to over time, run after run.

python
with open("log.txt", "a") as file:
    file.write("Second run started\n")

Try It Yourself: Write a small script that creates a file called notes.txt, writes three lines of your choice into it using write mode, then opens it again in read mode and prints each line, stripped of its trailing newline.

Lesson 8.2 — Reading & Writing CSV Files

Plain text files are fine for logs and notes, but the bread and butter of data engineering is the CSV file — comma-separated values, the format nearly every export from nearly every system eventually lands in. Let's learn to handle it properly.

Python has a built-in csv module, ready to use without installing anything. Let's first create a small sample file to work with, the same way real messy exports tend to look:

python
import csv

with open("orders.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["order_id", "customer_name", "total"])
    writer.writerow(["10432", "Priya Shah", "149.50"])
    writer.writerow(["10433", "Raj Kumar", "89.99"])
    writer.writerow(["10434", "Amit Verma", ""])

Notice that newline="" argument — it's a small, easy-to-forget detail that stops CSV files from getting extra blank lines inserted on some systems. It's not something to memorize deeply, just something to recognize and include whenever you're writing a CSV.

Now let's read it back, the way you'll do constantly in real work:

python
import csv

with open("orders.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(row)
Output / Note

{'order_id': '10432', 'customer_name': 'Priya Shah', 'total': '149.50'} {'order_id': '10433', 'customer_name': 'Raj Kumar', 'total': '89.99'} {'order_id': '10434', 'customer_name': 'Amit Verma', 'total': ''}

csv.DictReader reads each row and hands it to you as a dictionary, using the first row of the file — the header row — as the keys automatically. This is exactly the "list of dictionaries" shape you met back in Chapter 4, and it's genuinely how most real tabular data ends up looking once it's loaded into Python.

Look closely at that last row: 'total': '' — an empty string, not a proper number. This is the type-conversion lesson from Chapter 2, showing up again in its most common real form. Every single value read from a CSV file is text, always, even the ones that look like numbers, and some of them might be missing entirely. Let's handle that properly:

python
import csv

with open("orders.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        total_text = row["total"]
        if total_text == "":
            print(f"{row['customer_name']}: missing total — skipping")
            continue
        total = float(total_text)
        print(f"{row['customer_name']}: ${total}")
Output / Note

Priya Shah: $149.5 Raj Kumar: $89.99 Amit Verma: missing total — skipping

That combines nearly everything from this whole course so far — reading a file, looping through rows, checking for a problem, converting a type safely, and skipping bad data on purpose rather than crashing. This is genuinely close to real, working pipeline code.

Writing rows back out from dictionaries works with csv.DictWriter, the natural counterpart:

python
import csv

cleaned_orders = [
    {"order_id": "10432", "customer_name": "Priya Shah", "total": "149.50"},
    {"order_id": "10433", "customer_name": "Raj Kumar", "total": "89.99"},
]

with open("orders_clean.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["order_id", "customer_name", "total"])
    writer.writeheader()
    writer.writerows(cleaned_orders)

fieldnames tells the writer what your header row and column order should be, and .writeheader() writes that row before the data. .writerows() writes every dictionary in the list as one row each, matching each key to the right column automatically.

Try It Yourself: Create a CSV file called products.csv with columns product_id, name, and price, and three rows of made-up data — including one row with an empty price. Read it back with csv.DictReader, and print each product's name and price, printing "Price missing" instead for the one with no price.

Lesson 8.3 — Reading & Writing JSON Files

CSV is great for flat, table-shaped data — rows and columns. But not all data is that simple. APIs, in particular, almost always send back a different shape: nested, with lists inside dictionaries inside dictionaries. That shape has its own file format, called JSON, and it maps onto Python's dictionaries and lists almost perfectly.

Let's create a sample JSON file:

python
import json

customer = {
    "id": 10432,
    "name": "Priya Shah",
    "is_priority": True,
    "orders": [149.50, 89.99, 24.00]
}

with open("customer.json", "w") as file:
    json.dump(customer, file, indent=2)

That indent=2 argument makes the saved file nicely readable, with proper spacing — genuinely worth including any time you write JSON for a human to potentially look at later. Open customer.json in VS Code now and take a look — you'll see it looks almost exactly like the Python dictionary we started with.

Reading it back is just as direct:

python
import json

with open("customer.json", "r") as file:
    data = json.load(file)

print(data["name"])
print(data["orders"])
print(sum(data["orders"]))
Output / Note

Priya Shah [149.5, 89.99, 24.0] 263.49

Notice something genuinely nice about JSON, compared to CSV: json.load() handed us back a real dictionary, with real types already in place — is_priority came back as an actual Python True, not the text "True" you'd get from a CSV. JSON preserves types across the save and load, which CSV simply can't do, since a CSV file is just plain text underneath.

Real JSON files, especially from APIs, usually hold a list of records rather than just one:

python
import json

customers = [
    {"id": 101, "name": "Priya Shah", "orders": [149.50, 89.99]},
    {"id": 102, "name": "Raj Kumar", "orders": []},
]

with open("customers.json", "w") as file:
    json.dump(customers, file, indent=2)

with open("customers.json", "r") as file:
    data = json.load(file)

for customer in data:
    total = sum(customer["orders"])
    print(f"{customer['name']}: ${total}")
Output / Note

Priya Shah: $239.49 Raj Kumar: $0

Look at that — nested lists inside dictionaries inside a list, and it all just works, matching the exact shapes you've been practicing with since Chapter 4. That's why JSON and Python get along so naturally, and why it's the format you'll meet constantly once we start pulling data from real APIs, later in this course.

Try It Yourself: Create a dictionary representing one pipeline run, with keys pipeline_name, status, and a nested list called errors (which can be empty). Save it to a file called run_summary.json with proper indentation, then read it back and print a sentence summarizing the run, including how many errors it had using len().

Lesson 8.4 — Hands-On: Build a CSV Report Script

Let's bring this chapter together into a genuinely useful tool — one you'll actually recognize the shape of again later in this course, once we start building real pipelines. A CSV report script that can look at any CSV file and tell you, at a glance, whether it's safe to trust.

First, let's create a deliberately messy sample file to test against:

python
import csv

with open("sample_data.csv", "w", newline="") as file:
    writer = csv.writer(file)
    writer.writerow(["order_id", "customer_name", "total"])
    writer.writerow(["10432", "Priya Shah", "149.50"])
    writer.writerow(["10433", "", "89.99"])
    writer.writerow(["10434", "Amit Verma", ""])
    writer.writerow(["10435", "Raj Kumar", "45.00"])

Your task: write a script that reads any CSV file and reports:

  1. How many rows it has.
  2. How many columns it has, and their names.
  3. How many missing (empty) values are in each column.

Here's the shape to build from:

python
import csv

def profile_csv(file_path):
    with open(file_path, "r") as file:
        reader = csv.DictReader(file)
        rows = list(reader)

    column_names = reader.fieldnames
    row_count = len(rows)

    print(f"File: {file_path}")
    print(f"Rows: {row_count}")
    print(f"Columns ({len(column_names)}): {column_names}")

    for column in column_names:
        missing_count = 0
        for row in rows:
            if row[column] == "":
                missing_count += 1
        print(f"  {column}: {missing_count} missing")

profile_csv("sample_data.csv")
Output / Note

File: sample_data.csv Rows: 4 Columns (3): ['order_id', 'customer_name', 'total'] order_id: 0 missing customer_name: 1 missing total: 1 missing

Notice we turned the reader into a proper list with list(reader) right away — that's because a CSV reader can normally only be looped through once, and we needed to loop through the rows multiple times: once to count them, and again for every column while checking for missing values. Storing it as a list up front sidesteps that limitation cleanly.

Look at what you've built: hand this function any CSV file path, and in three lines of output, it tells you exactly how trustworthy that file is, before you commit to processing it further. This is genuinely the first real check a working data engineer runs on almost any new file that shows up — profile it before you trust it.

Try It Yourself: Extend profile_csv to also report, for the total column specifically, how many of the non-empty values fail to convert to a float — using a try/except from Chapter 7 around the conversion. Test it by adding a row to sample_data.csv with the text "unknown" in the total column instead of a number.