Module 1: Project Structure, Packaging, Dependency Management

What you'll learn in this module

By the end of this module, you'll be able to:

Explain the real difference between a script and an installable Python package — and why that difference matters the moment your code leaves your own laptop
Structure a project using the src/ layout, and explain why this layout exists instead of just being a style preference
Write a pyproject.toml that declares your package, pins its dependencies, and defines a real command-line entry point
Use pip install -e . correctly, and recognize the specific kind of confusing bug that shows up when you forget it
Work through your first AI-assisted coding round using the closed loop — including, if this is new to you, a careful walkthrough of exactly how to prompt, review, and iterate with an AI coding assistant

Let's begin.

Let's start with a question

You write a script. It works. You email it to a teammate — or you push it to a shared repo, and they pull it down.

They run it. It breaks.

Nothing about the code changed. So what did?

Almost always, the honest answer is: your script was never really independent of your machine. It worked because of something true only on your laptop — a file sitting in just the right folder, a package you installed once and forgot about, a path that happened to resolve because of where you happened to be standing when you typed the command.

This module is about closing that gap. Not by writing better logic — the logic in this module's lab is almost trivially simple on purpose. It's about making sure your code carries its own truth with it, instead of quietly depending on the accident of your machine.

Why this matters in data engineering

Here's a scenario you will absolutely run into, probably in your first few months on any real data team.

You build a pipeline. It works great, on your laptop. Then:

It needs to run on a teammate's machine too, and suddenly doesn't.
It needs to run inside a Docker container, where your laptop's folder structure doesn't exist at all.
It needs to run on a schedule, at 3am, with nobody around to notice if it silently uses the wrong version of a dependency.
Six months from now, someone — maybe you — needs to install this thing fresh, and has no idea what it actually depends on.

Ask yourself: if someone handed you your own code right now, with no memory of writing it, could you get it running in five minutes? If the honest answer is "only if they ask me first," this module is for you.

The core idea: a script vs. a package

A script is something you run by typing python some_file.py, from inside the one folder where it happens to work.

A package is something you can install — with pip install . or pip install -e . — and then run, or import, from anywhere. It doesn't care where you're standing when you run it. It carries its own map of what it needs.

That difference sounds small. It is not small. It's the difference between "runs on my machine" and "runs."

`src/` layout vs. flat layout

There are two common ways to lay out a Python project. Let's look at both.

Flat layout — your package code sits right next to your project's other files, like setup.py, README.md, tests, and so on, all in the same top-level folder.

src/ layout — your actual package code lives one level deeper, inside a src/ folder: src/yourpackage/....

Why would anyone add an extra folder level on purpose? Here's the honest answer: it prevents a specific, sneaky bug. Without a src/ folder, Python can sometimes accidentally import your local, uninstalled code instead of the version you actually installed — because your current folder is sitting right there on the import path, ready to be picked up by accident. With everything nested under src/, that accidental path simply isn't there anymore. Python is forced to import the version you actually installed, on purpose.

`pyproject.toml`: the modern way to describe a package

Older Python projects used a file called setup.py to describe themselves — what the package is called, what it depends on, how to install it. Modern Python projects use pyproject.toml instead. Same job, cleaner format, and it's now the standard the whole ecosystem has settled on.

Inside it, you'll declare:

What your package is called, and what version it's at
What it depends on — and depends on at a specific version, not just "whatever's newest today." (Ask yourself: if you don't pin a version, and a dependency ships a breaking change next month, when do you find out? Answer: whenever your pipeline breaks, at the worst possible time, with no warning.)
Its CLI entry point — this is what lets someone type salesreport generate instead of python some/deeply/nested/file.py

Editable installs: `pip install -e .`

One more idea worth sitting with. When you're actively developing a package, you don't want to reinstall it every time you change a line of code. That's what pip install -e . is for — the -e means "editable." It installs your package in a way that always points back at your source files, live. Change the code, and the installed version changes with it, instantly.

Without -e, you'd be stuck reinstalling constantly, or worse — not realizing you're testing against a stale, previously-installed copy while your actual edits sit there doing nothing. That's not a rare mistake. It is one of the most common causes of "I fixed this, I swear" bugs — where an engineer edits code, re-runs a test, sees the same failure, and concludes the fix didn't work, when actually the fix was never running at all.

One more thing worth knowing before you actually run this: on a modern system, pip install (with or without -e) usually won't let you install straight into your system's Python at all — and that's by design, not a bug you need to work around. The fix is a virtual environment: an isolated, self-contained copy of Python just for this one project. You'll set one up in the lab below, and it's worth understanding why it exists rather than just typing the commands — every real Python project uses one, because different projects often need different, sometimes conflicting versions of the same dependency, and a virtual environment keeps them from stepping on each other.

Manual lab: turn a script into a real package

Getting the lab files

The starter code and full solution for this module are provided as a download from the course platform, named module-1-materials.zip. This is not something you clone from a course-wide repository — it's yours to bring into your own production-python repo, the same way you brought in your original starter files during setup.

Download module-1-materials.zip from the course platform.
Inside your own repo (~/courses/pp4de from setup), create a dedicated folder for module lab work, kept separate from your actual capstone code:
```
bash
cd ~/courses/pp4de
mkdir -p course-materials
```

Copy the zip in and extract it:


bash
cp /mnt/c/Users/yourname/Downloads/module-1-materials.zip course-materials/
cd course-materials
unzip module-1-materials.zip

You should now have course-materials/module-1/starter/ and course-materials/module-1/solution/.

Commit this to your repo, same as always:


bash
cd ~/courses/pp4de
git add course-materials/module-1
git commit -m "Add Module 1 lab materials"
git push

Why a separate course-materials/ folder, instead of mixing this into src/? Because src/ is reserved for your real capstone code — the thing you'll actually walk through in an interview. Lab practice code and capstone code should never end up tangled together in the same folder, for the same reason your interview drills and your capstone stay separate: mixing them makes it unclear later which code was practice and which was the real thing.

The scenario

You've inherited a small internal tool from a teammate who left the company. It's called sales_report.py, and it prints total revenue per store from a CSV of daily sales. Your manager wants to start using this tool from other places — a scheduled job, a teammate's machine, and eventually a Docker container. Right now, none of that is possible: the tool only works if you happen to be standing in exactly the right folder when you run it. Your job in this lab is to fix that, without changing what the tool actually does.

Reproduce the problem first

Before fixing anything, confirm the problem is real and see it fail with your own eyes. This matters — debugging a problem you've actually watched happen is a different skill than debugging one someone described to you.

Open a terminal and go into the starter folder:
```
bash
cd course-materials/module-1/starter
```
Run the script exactly as given:
```
bash
python3 sales_report.py
```
You should see revenue printed per store — the tool works, right now, from this exact spot.
Now move up one directory, and run the exact same script from there, using its path:
```
bash
cd ..
python3 starter/sales_report.py
```
Watch it fail with a FileNotFoundError. Nothing about the script changed between steps 2 and 3 — only where you were standing when you ran it.
Open starter/sales_report.py and starter/helpers.py and find the line responsible. You're looking for a hardcoded path that only resolves correctly from one specific working directory.

You've now reproduced the exact bug this lab exists to fix, and you know precisely where it lives in the code.

Your task

Turn this into a real, installable Python package. Specifically:

Restructure the code into a src/ layout — a proper package, not a loose collection of files.
Write a pyproject.toml that declares the package, pins its dependencies to specific versions, and defines a CLI entry point.
Give it a real command-line interface — something like:
```
bash
salesreport generate --input path/to/some.csv
```
instead of a hardcoded path baked into the script.
Install it with pip install -e ., and prove to yourself that it now works from any directory — the same test that broke the starter code (steps 3-4 above) should now succeed instead.

The actual reporting logic (read a CSV, total revenue per store) should not need to change at all. This lab isn't about the logic — it's entirely about the structure around it. If you find yourself rewriting the business logic, you've gone further than the lab is asking for.

A question worth asking before you look at the solution

We've said the reporting logic shouldn't need to change — so if you compare the solution to the starter code and it looks different, that's worth pausing on, not skipping past.

Here's the honest answer: the actual data-processing logic — reading the CSV, totaling revenue per store — doesn't change at all. Compare starter/helpers.py to solution/src/salesreport/report.py line by line, and the only difference you'll find is type hints. Same loop, same variable names, same logic.

What does genuinely change is the command-line interface — and that's not an accident or a contradiction, it's because step 3 of your task explicitly asked for a real CLI. Going from a bare if __name__ == "__main__": block to a proper click-based command is new work, on purpose. The module never claimed the interface would stay the same — only the logic underneath it.

So: logic moved, untouched. Structure and interface, rebuilt on purpose. Keep that distinction in mind as you read the solution below.

Full worked solution

The complete solution lives in course-materials/module-1/solution/:

solution/
├── pyproject.toml
├── src/
│   └── salesreport/
│       ├── __init__.py
│       ├── cli.py          # only this file knows about the command line
│       └── report.py       # the actual logic - knows nothing about argv
└── sample_data/
    └── sales_sample.csv

A few things worth noticing once you compare it to your own attempt:

report.py has no idea it's being called from a CLI. It just takes a path in, and returns data out. That separation is deliberate — it keeps the logic easy to test and easy to reuse from somewhere that isn't a command line at all.
cli.py is the only file that imports click. If you ever swapped out the CLI framework entirely, only this one file would need to change.

Verify it, step by step

Don't just read the solution — install it and prove to yourself it actually fixes the original bug.

From inside solution/, create and activate a virtual environment first. Modern systems block installing packages straight into your system Python — this isn't something to work around, it's standard practice for any real Python project, since different projects often need different versions of the same dependency:
```
bash
cd course-materials/module-1/solution
python3 -m venv .venv
source .venv/bin/activate
```
Your terminal prompt should now start with (.venv) — that's your confirmation you're working inside the isolated environment.
Now install in editable mode:
```
bash
pip install -e .
```
Confirm the command exists:
```
bash
salesreport --help
```
You should see a generate subcommand listed.
Run it from inside the solution/ folder first, as a baseline:
```
bash
salesreport generate --input sample_data/sales_sample.csv
```
You should see the same revenue-by-store output as the starter script produced.
Now the real test — the one the starter code failed. Move somewhere completely unrelated, and run the exact same command, using a full path to the CSV. Stay in the same terminal session — a fresh terminal window wouldn't have this virtual environment activated, and salesreport would appear to stop working for a reason that has nothing to do with what this test is actually checking:
```
bash
cd /tmp
salesreport generate --input ~/courses/pp4de/course-materials/module-1/solution/sample_data/sales_sample.csv
```
(Adjust the path if your repo lives somewhere else.) This should succeed, with identical output — proving the fix works regardless of where you're standing, unlike the starter code.
One more check worth doing: confirm salesreport generate --help works from /tmp too, not just from inside the project. A real installed CLI shouldn't care where you invoke it from.

If step 5 succeeds, you've verified the actual thing this module set out to teach — not just read about it.

AI-assisted round

If this is your first time doing a structured AI-assisted coding round — not just asking a chatbot for a snippet, but actually working through the closed loop — go slowly here. The steps below are more detailed than they'll need to be in later modules, on purpose, since the habit needs to be built once, carefully, before it can be done quickly.

Make sure you've completed the assistant setup and the small demo exercise described in your AI assistant setup guide before continuing — you should already have either Copilot's agent mode or Claude Code running and confirmed working.

Step 1: Open your assistant against the right project

Open your solution/ folder (or your own working version of it, if you completed the task yourself) in VS Code. Make sure your assistant is pointed at this project specifically — not a blank chat window with no file access. If you're using Copilot, this means the chat panel is open inside this VS Code window, in Agent mode. If you're using Claude Code, this means you've run claude from inside this project's folder, or opened the extension with this folder active.

Step 2: Use this exact prompt

Copy this prompt in as-is the first time, so you have a clean, repeatable starting point:

Output / Note

"Add a validate subcommand to this CLI that checks a sales CSV file for missing or malformed rows before generating a report. A row is invalid if it's missing store_id or quantity, or if quantity isn't a valid number. Print how many rows were skipped and why. Write a small test for this, and run it before telling me you're done."

Notice the last sentence: "run it before telling me you're done." This is not optional politeness — it's what makes the closed loop actually work instead of collapsing into vibe coding. You are explicitly requiring evidence, not just a claim.

Step 3: Watch what it proposes — don't accept yet

Whatever your assistant proposes, resist the urge to click "accept" right away, even if it looks reasonable. Read through the plan or the diff first, and ask yourself:

Where did it put the new code? Is it inside your package's src/ folder, alongside the other modules — or did it create a new file somewhere else, like the project root?
Did it actually write and run a test, the way you asked — or did it just claim the change works?
Does the validation logic match exactly what you asked for (missing store_id, missing quantity, non-numeric quantity) — or did it quietly add, skip, or reinterpret part of the requirement?

Step 4: Let it actually run

If you're using agent mode or Claude Code correctly, the assistant should be able to execute the test itself and show you real output — not just describe what it expects to happen. If it stops short of actually running anything, explicitly ask it to: "Please run the test now and paste the real output." Do not accept a description of expected behavior as a substitute for seeing it actually run.

Step 5: Review the diff like a pull request

Once you've seen real test output, go through the actual code change one more time, line by line, the way you would for a teammate's pull request before approving it. This is the step most people skip when they're new to AI-assisted coding, and it's the single most important one.

Step 6: Check it against this module's known failure pattern

Two failure patterns are especially common for exactly this kind of task. Check for both, deliberately, in the diff you're reviewing:

Pattern 1 — layout inconsistency. An assistant extending an existing src/-layout package will sometimes add the new command as a file outside src/salesreport/ — for instance, creating a top-level validate.py next to pyproject.toml, instead of inside the package. It often still runs locally, because your editable install's import machinery can be more forgiving than you'd expect from inside the project folder — which is exactly what makes this easy to miss. The fix works. The structure is now inconsistent, and the exact bug this module exists to prevent has quietly come back.

Pattern 2 — inventing options for the wrong library version. Assistants frequently generate click code using an option or decorator pattern from a newer or older click release than the one actually pinned in your pyproject.toml. It can look completely correct, read fine, and still fail at runtime with an error that has nothing to do with your actual logic. If your assistant's proposed code uses any click feature you don't recognize, check it against the version pinned in pyproject.toml before accepting.

Here's what pattern 1 tends to look like in practice, if you want a concrete before-and-after to compare against what your own assistant proposes:


diff
+ # validate.py  (created at the project root, next to pyproject.toml)
+ import click
+ from salesreport.report import read_sales_rows
+
+ @click.command()
+ @click.option("--input", "input_path", required=True, type=click.Path(exists=True))
+ def validate(input_path):
+     rows = read_sales_rows(input_path)
+     skipped = sum(1 for r in rows if not r.get("store_id") or not r.get("quantity"))
+     click.echo(f"Skipped {skipped} invalid rows")


diff
  # src/salesreport/cli.py
  from salesreport.report import read_sales_rows, total_revenue_by_store
+ from validate import validate as validate_cmd

  @cli.command()
  ...

+ cli.add_command(validate_cmd)

Run this locally, right after the change, and it works — because the current working directory happens to be the project root, where validate.py conveniently sits. Install the package properly, or run it from a different directory, and the import breaks, because validate.py was never actually part of the installed package — it's not under src/, so it was never included in the build.

Step 7: Fix it if you found it, and write your guardrail

If your assistant made either mistake, ask it to fix the file location or the library usage explicitly, and confirm the test still passes afterward. Then write down one line in your AI guardrail log — this is the actual deliverable of this round, not the working validate command itself. Something like:

Output / Note

Module 1 guardrail: Before accepting any AI-proposed new file, check where it physically landed. If it's not inside src/<package>/, reject the diff and ask for it again, explicitly naming the correct location.

If your assistant got it right the first time — that happens too, and it's worth noting in your log as well. Write down what you checked for and confirmed clean, not just the mistakes you caught. That's still evidence of the discipline, and it's worth having in the log either way.

Common mistakes

Watch for these. They're common enough that naming them in advance is worth more than discovering them the hard way.

Mixing flat and src/ layout in the same repo. Usually happens gradually — a project starts flat, someone partially migrates it to src/, and now some imports resolve one way and some resolve another, depending on exactly how the package was installed. Pick one layout and commit to it fully; a half-migration is worse than either layout alone.
Forgetting -e and "fixing" phantom bugs. If you install without -e, your changes to the source code stop taking effect — you're now testing a frozen snapshot from whenever you last ran pip install . This produces a very specific, confusing kind of bug: you fix something, test it, and it's still broken, because you're not actually running the code you just edited.
Unpinned dependencies. click with no version number means "whatever's newest whenever this happens to get installed." That's fine right up until a new release changes behavior out from under you, with no warning, on a machine you're not even watching.
A CLI file that knows too much. If your CLI layer starts containing actual business logic — not just argument parsing and a call out to the real logic — you've lost the separation that makes testing and reuse straightforward. The CLI file's job is to translate a command line into a function call. Nothing more.

Capstone tie-in

This is where your real capstone repo takes its first shape. Not the practice repo. Not course-materials/. Your actual pp4de project — the one you'll carry all the way to Module 9.

You're about to do the same thing you just practiced in the lab. But this time, there's no starter code to fix. Your capstone repo right now only has infra/, SPEC.md, README.md, and the setup docs — no package exists yet. You're building it fresh, using exactly what you just learned.

Step 1: Open your capstone repo in VS Code

Make sure you're looking at your real pp4de project, not course-materials/module-1. Check the title bar at the top of the VS Code window — it should say pp4de [WSL: Ubuntu], not solution [WSL: Ubuntu].

Step 2: Build the `src/` layout, using the Explorer panel

In VS Code's Explorer:

Right-click the top-level pp4de folder → New Folder → name it src.
Right-click src → New Folder → name it pipeline. This is your package name — the capstone's ingestion service.
Right-click pipeline → New File → name it __init__.py. Leave it empty.

You should now see:

pp4de/
├── infra/
├── src/
│   └── pipeline/
│       └── __init__.py
├── SPEC.md
└── ...

Step 3: Create `pyproject.toml`

In VS Code's Explorer:

Right-click the top-level pp4de folder → New File → name it pyproject.toml.
Type or paste this in:


toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "pipeline"
version = "0.1.0"
description = "Production-grade ingestion and transformation service"
requires-python = ">=3.11"
dependencies = [
    "click==8.1.7",
]

[project.scripts]
pipeline = "pipeline.cli:cli"

[tool.hatch.build.targets.wheel]
packages = ["src/pipeline"]

This is the same shape of file you just wrote in the lab — same sections, same idea. Only the names changed: pipeline instead of salesreport.

Step 4: Build a CLI stub — no real logic yet, on purpose

Right now, this command doesn't need to actually ingest anything. That logic comes in later modules. All it needs to do today is exist, accept the right arguments, and print what it would do. You're proving the shape is right before you pour real logic into it.

In VS Code's Explorer:

Right-click pipeline (inside src/) → New File → name it cli.py.
Type or paste this in:


python
"""Command-line entry point for the ingestion pipeline.

This is a stub for now - it doesn't ingest anything yet. That logic
arrives in later modules. Right now, this file's only job is to prove
the package, the CLI entry point, and the installed command all work
correctly end to end.
"""

import click


@click.group()
def cli() -> None:
    """pipeline - production-grade ingestion and transformation service."""


@cli.command()
@click.option(
    "--date",
    required=True,
    help="Date to ingest records for, in YYYY-MM-DD format.",
)
def ingest(date: str) -> None:
    """Ingest records for a given date. Stub only, for now."""
    click.echo(f"Would ingest records for {date}. Not implemented yet.")


if __name__ == "__main__":
    cli()

Step 5: Install it and prove it works — the one mandatory terminal step

Open VS Code's integrated terminal (**Ctrl+**, or **View → Terminal**). Make sure you're at the root of pp4de, not inside src/`.

Same reasoning as the lab — a virtual environment first, not straight into the system Python:


bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Then confirm it actually works:


bash
pipeline --help
pipeline ingest --date 2026-07-01

You should see the ingest command listed, and running it should print the "Would ingest records for 2026-07-01..." message. That's the whole goal for this step — not real ingestion, just proof the structure, the package, and the entry point all genuinely work together.

Step 6: Create your AI guardrail log

This is where the guardrail you wrote earlier, in this module's AI-assisted round, actually gets recorded permanently — not just left sitting in your notes.

In VS Code's Explorer:

Right-click the top-level pp4de folder → New File → name it AI_GUARDRAIL_LOG.md.
Add a short intro line explaining what this file is, then a ## Module 1 heading, followed by the guardrail you wrote earlier in this module — what the assistant actually did, what you caught, and the one-line rule you're adopting going forward.

This file lives at the root of your repo, next to SPEC.md — it's a running log you'll keep adding to, one section per module, all the way through Module 9. Don't rewrite it each time; just add a new heading and entry underneath the last one.

Step 7: Commit and push — through VS Code's Source Control panel

No terminal git commands needed here — this is a good moment to use VS Code's built-in Source Control UI instead, since it's just as capable and keeps you in the same rhythm as the rest of this walkthrough.

Click the Source Control icon in the left sidebar (it looks like a branching line — you'll see a number badge on it showing how many files changed).
You should see src/, pyproject.toml, AI_GUARDRAIL_LOG.md, and possibly a .venv folder listed under Changes. Check first: .venv should not appear here. If it does, your .gitignore isn't catching it — stop and fix that before committing (there should already be a .venv/ line in pp4de/.gitignore from setup; if it's missing, add it now).
Hover over Changes, and click the + icon that appears to stage everything — or stage files one at a time if you'd rather review each one first.
Type a commit message in the box at the top, something like: Module 1: package structure, pyproject.toml, CLI stub
Click the ✓ Commit button (or Commit from the ... menu).
Click Sync Changes (or Push, depending on your VS Code version) to send this commit to your production-python repo on GitHub.
Refresh your repository page on GitHub, and confirm src/ and pyproject.toml are there.

This is your first real capstone commit. Everything from Module 2 onward builds directly on top of this structure.

Check row 1 off your SPEC.md checklist. Nine to go.

Interview drill

Every question below follows the same pattern. First, the question. Then, what the interviewer wants to hear. Then, a junior engineer's answer. Then, a strong senior answer.

Read the junior answer first. Notice what it's missing. That gap teaches you more than the senior answer alone.

Recall

Question: "What's the real difference between pip install . and pip install -e .? Give me a real example where this difference caused a confusing bug."

What the interviewer wants to hear: not just the definition. A real, specific story. Something that shows this has actually confused you or a teammate before. Not a textbook answer.

Junior answer: "-e means editable install. It lets you edit the code, and the changes apply right away. Without -e, you'd need to reinstall every time you change something."

(This is correct. But it's generic. No real story. No sense of why this actually matters.)

Senior answer: "A plain pip install . copies your package at that one moment. It's a snapshot. pip install -e . links the install back to your real source files. So changes apply right away — no reinstall needed. Here's where this actually bites people: I once watched an engineer waste twenty minutes. They were sure their bug fix wasn't working. They'd fix the code, run the test, see the same failure. Fix it again. Same failure. Turns out — they'd installed without -e days earlier and forgot. Every test ran against an old, frozen copy. The fix was never even running. My rule: use -e for anything I'm actively building locally. Use a plain install for anything meant to be a fixed, final build — like testing the real thing before it goes to production."

Debugging

Question: shown as a real code snippet and a real incident story — not as an abstract question. The learner sees only this. No explanation of what's wrong yet:


python
# shipctl/pricing.py
from shipctl.rates import get_rate

def calculate_shipping(weight_kg: float, zone: str) -> float:
    rate = get_rate(zone)
    return round(weight_kg * rate, 2)

shipctl/
├── shipctl/
│   ├── __init__.py
│   ├── pricing.py
│   └── rates.py
├── setup.py
└── rates.py          # <-- also exists at repo root

"A teammate says they fixed a bug last week. They updated the zone rate table in rates.py. They ran pip install -e . after the fix. Locally, tests pass. But in the deployed container, the old rates are still being used. Why?"

What the interviewer wants to hear: correctly naming this as an import-resolution problem, not a caching problem (a common wrong first guess). Knowing how to actually check which file gets loaded. Proposing a real structural fix — not just deleting the extra file and moving on.

Junior answer: "There are two rates.py files — one inside the package, one at the repo root. The teammate probably edited the wrong one. I'd delete the root-level file so there's only one."

(This finds the duplicate and fixes the symptom. But it doesn't explain why two files could satisfy the same import in the first place. And it doesn't propose anything to stop this from happening again.)

Senior answer: "This smells like an import-resolution problem, not a caching problem. First, I'd actually confirm which file is being loaded — not guess. I'd run python -c "import shipctl.rates; print(shipctl.rates.__file__)". Given this layout, there are two rates.py files that could both satisfy from shipctl.rates import get_rate. Depending on how the package was installed — editable install versus a built wheel — and what's on sys.path in each environment, either file could win. My guess: the teammate edited the stray root-level file. That happened to be the one their local setup picked up. But it's not the one that ends up in the built container image. The real fix isn't just deleting the duplicate. It's moving to a src/ layout. That makes this exact kind of duplication structurally impossible — there's no ambiguous root-level spot for a stray file to hide in. I'd also add a CI check that fails the build if the editable install's resolved package path doesn't match the expected src/ location. That way, this can't silently happen again in some other module."

AI-review

Question: the learner is shown this diff, with no explanation of what's wrong yet. They're asked what's wrong with it, and what they'd tell the assistant to fix:


diff
+ # validate.py  (created at the project root, next to pyproject.toml)
+ import click
+ from salesreport.report import read_sales_rows
+
+ @click.command()
+ @click.option("--input", "input_path", required=True, type=click.Path(exists=True))
+ def validate(input_path):
+     rows = read_sales_rows(input_path)
+     skipped = sum(1 for r in rows if not r.get("store_id") or not r.get("quantity"))
+     click.echo(f"Skipped {skipped} invalid rows")


diff
  # src/salesreport/cli.py
  from salesreport.report import read_sales_rows, total_revenue_by_store
+ from validate import validate as validate_cmd

  @cli.command()
  ...

+ cli.add_command(validate_cmd)

What the interviewer wants to hear: correctly spotting this as a packaging problem, not a logic problem. The validate function itself works fine. The problem is where it lives. A strong answer names the exact fix — not just "this looks off."

Junior answer: "Looks fine to me. The function does what was asked — it checks for missing fields and counts skipped rows. I'd probably just run it and see if it works."

(This runs it locally, sees it work, and completely misses the problem — because it does work locally. That's exactly what makes this failure pattern dangerous.)

Senior answer: "The logic inside validate is fine. The real bug is where the file was created. It's sitting at the project root, outside src/salesreport/. That means it was never actually made part of the installed package. Run this locally, right after the change, and it'll work — because the current folder happens to be the project root, so Python can still find validate.py by accident. But install this package properly, or run it from anywhere else, and the import in cli.py breaks — from validate import validate as validate_cmd — because validate.py was never part of the built package at all. I'd tell the assistant exactly what to do: move this file into src/salesreport/. Rename the import to from salesreport.validate import validate as validate_cmd. Then re-run it from a folder other than the project root, to actually prove the fix works — the same check I'd run for any packaging change."

Judgment

The judgment round works like a real interview. It's a back-and-forth, not one question with one answer. Below is a full example conversation. After that, you'll see one strong answer — the kind that could satisfy an interviewer without needing every round of pushback.

Interviewer: "You join a team that maintains an inventory sync service. The code uses a flat layout — no src/ folder. It uses setup.py, not pyproject.toml. Dependencies have no version pins. It's been in production for two years with no major problems. Your tech lead says, 'if it's not broken, don't touch it — just focus on your ticket.' What do you do?"

Candidate (asking first): "Before I answer — a couple of things would change my thinking. How big is my actual ticket? A small fix, or something that touches a lot of the code? And what does test coverage look like right now?"

Interviewer: "Good questions. The ticket is a medium one — you're adding support for a new data source. It touches a real chunk of the sync logic. Test coverage is thin. Maybe 40%. Mostly happy-path tests. Not much coverage for edge cases or failures."

Candidate (first answer): "Given that, I'd leave the project structure alone for this ticket. A full move to src/ layout is a separate, riskier piece of work. Mixing it into a feature change makes the diff harder to review. It also makes it harder to roll back cleanly if something breaks. I'd focus on shipping the feature."

Interviewer (pushback): "But you just told me you're worried about low test coverage and risk. If the structure is already messy, doesn't adding more code just make the cleanup harder later? Why not fix it now, while you're already in there?"

Candidate (round 1): "Fair challenge. But let me separate two things: 'messy' and 'risky to change.' The flat layout isn't causing bugs today. It's a maintenance problem, not a correctness problem. What is risky is doing a big structural change on a two-year-old production system, with only 40% test coverage and no safety net. I'd rather ship the feature cleanly first. Then raise the structure issue separately, as its own reviewed piece of work — ideally once coverage improves enough to safely check a migration."

Interviewer (second pushback): "Okay. But this new data source needs a new module. Doesn't that just mean you're adding one more file to the same messy structure? Doesn't that make the eventual migration bigger?"

Candidate (round 2): "That's actually a good moment to do something small and low-risk. I'd put the new code in a clearly named, self- contained module — even inside the current flat layout. Not scattered across old files. That doesn't fix the structure today. But it means this new code moves as one clean unit later, instead of needing to be untangled first. I'd also open a ticket documenting the migration — roughly how much work it is, and what needs to be true first, like better test coverage. That way it doesn't just become an unspoken complaint that nobody ever writes down."

Interviewer (final challenge): "Convince me this isn't just kicking the can down the road."

Candidate (final defense): "It would be kicking the can if I never sequenced it. But I'm not saying 'never.' I'm saying 'not tangled up with unrelated feature work, on a system where we don't yet have the test coverage to safely check a big structural change.' Concretely: ship the feature, scoped tightly. Right after, do two small, low-cost wins — like pinning the dependencies, which is quick and safe on its own. Then put the structural migration on the team's radar as a scheduled, reviewed piece of work — not some vague 'someday.' That's a sequence. Not an avoidance."

Model strong answer — one answer, given upfront, that would satisfy most interviewers without needing every round of pushback above:

Output / Note

"I'd ask two things first: how big is this ticket, and what does test coverage look like right now? Both affect how much risk a structural change would add. Let's say this is a medium feature ticket, on a codebase with thin test coverage. I'd scope my work to just the feature. I'd deliberately not fold in a src/ layout migration — that's a separate, higher-risk change. Doing both at once makes the diff harder to review, and harder to roll back if something breaks. That said, 'don't touch it' doesn't have to mean 'ignore it forever.' I'd do a few small, genuinely low-risk things alongside the feature: put the new code in a clearly isolated module, so it's easy to move later. And pin the currently-unpinned dependencies — that's a quick, safe win on its own. Then I'd open a separate ticket for the real structural migration. I'd give it a rough scope, and a clear trigger — for example, 'revisit once test coverage is above 70%, so we can check the migration didn't quietly break anything.' That way, the concern doesn't just disappear because nobody wrote it down. But it also doesn't get tangled into a feature change where it adds risk without adding value to this specific ticket."

Module 1: Project Structure, Packaging, Dependency Management

What you'll learn in this module

Let's start with a question

Why this matters in data engineering

The core idea: a script vs. a package

src/ layout vs. flat layout

pyproject.toml: the modern way to describe a package

Editable installs: pip install -e .

Manual lab: turn a script into a real package

Getting the lab files

The scenario

Reproduce the problem first

Your task

A question worth asking before you look at the solution

Full worked solution

Verify it, step by step

AI-assisted round

Step 1: Open your assistant against the right project

Step 2: Use this exact prompt

Step 3: Watch what it proposes — don't accept yet

Step 4: Let it actually run

Step 5: Review the diff like a pull request

Step 6: Check it against this module's known failure pattern

Step 7: Fix it if you found it, and write your guardrail

Common mistakes

Capstone tie-in

Step 1: Open your capstone repo in VS Code

Step 2: Build the src/ layout, using the Explorer panel

Step 3: Create pyproject.toml

Step 4: Build a CLI stub — no real logic yet, on purpose

Step 5: Install it and prove it works — the one mandatory terminal step

Step 6: Create your AI guardrail log

Step 7: Commit and push — through VS Code's Source Control panel

Interview drill

Recall

Debugging

AI-review

Judgment

`src/` layout vs. flat layout

`pyproject.toml`: the modern way to describe a package

Editable installs: `pip install -e .`

Step 2: Build the `src/` layout, using the Explorer panel

Step 3: Create `pyproject.toml`