Course Introduction

Let's start with a question

Have you ever written Python code that worked perfectly on your laptop?

You ran it. It gave the right output. You felt good.

Then someone else ran the same code. It broke.

Or maybe you ran it again yourself, a week later. This time, it crashed halfway through. Or worse — it ran fine, but now your database has duplicate rows nobody asked for.

If this has happened to you, you are not alone. Almost every data engineer has a story like this. The difference between a junior engineer and a senior engineer is not that the senior engineer never writes buggy code. The difference is this: the senior engineer expects things to go wrong, and builds for it.

That is what this course teaches you. Before we go further, let's answer the questions you're probably already asking.

What is this course?

This is a hands-on, interview-oriented course on Production Python for Data Engineers.

Not a syntax course. Not "learn Python from scratch." You already know how to write Python. What you're about to learn is how to write Python the way a senior engineer does — code that survives contact with the real world, and that you can defend, out loud, in an interview.

Every module in this course follows a simple rhythm: learn one idea, practice it by hand, practice it again with an AI assistant, then defend it under interview-style questioning. One capstone project grows across all nine modules. By the end, you'll have a real, working system — not nine disconnected exercises.

What this course is not:

It's not a Python fundamentals course. If terms like functions, loops, or dictionaries are unfamiliar, this course will move too fast.
It's not a broad tools survey. We go deep on a small number of things that show up in almost every real data engineering job, instead of shallow on everything.
It's not a "watch and learn" course. You will write code, break it on purpose, and defend your decisions out loud. There's no passive way to get value from this course.

What will you take away from this course?

By the end, you will have:

A real capstone project, built by your own hands, module by module — not a tutorial you followed once and forgot. Something you can actually open and walk an interviewer through.
Proof you can work with AI without switching off your judgment. An AI guardrail log with at least nine real entries — real mistakes you caught, real fixes you made. This is quickly becoming one of the most interview-relevant things a candidate can show.
Fluency in how senior interviews actually work. Four question types — recall, judgment, debugging, and AI-review — practiced repeatedly, on unfamiliar scenarios, not memorized answers.
A working habit, not just knowledge. By the end, "will this break if I run it twice?" or "did I actually test the failure path?" should be questions you ask automatically, not things you remember to check when someone reminds you.
3-4 real behavioral interview stories, collected from your own work history as you go, so you're not scrambling to invent them the night before an interview.

Who is this course for?

This course is built for data engineers who already know Python, but haven't yet built production systems.

More specifically, you're a good fit if:

You're comfortable writing Python — functions, classes, basic error handling — but most of your code has been scripts, notebooks, or one-off analysis, not systems that run unattended in production
You're preparing for data engineering interviews at the mid-to-senior level, and want to practice the kind of questions those interviews actually ask
You've felt the gap between "I can write code that works" and "I can build something a team can rely on," and want to close it deliberately

This course is not a good fit if:

You're brand new to Python itself — this course assumes you can already write and read Python comfortably
You're looking for a broad survey of every data engineering tool (Spark, Airflow, dbt, and so on) — this course goes deep on core engineering discipline instead, using a small, focused toolset
You already have several years of production experience and are looking for advanced, specialized topics rather than a grounding in the fundamentals of production-quality code

Prerequisites

Here's what you need before you start:

Comfortable Python fundamentals. Functions, classes, basic exception handling, reading someone else's code without getting lost. You don't need to know testing, typing, or async — those are taught from scratch in this course.
Basic command-line comfort. You should be okay typing commands into a terminal, even if you're not an expert. We walk through everything step by step, including for learners new to Linux-style terminals.
Basic git familiarity. Cloning, committing, pushing. Again, we walk through this from scratch in setup — but if you've never touched git at all, expect setup to take a little longer than described.
No prior production experience required. That's the entire point of this course — if you already had it, you wouldn't need this.
A laptop that can run Docker. Windows, Mac, or Linux all work. Setup instructions assume you're starting from zero.

How is this course structured?

Format: a course introduction (this document), 9 skill modules, a capstone integration phase, and a final interview simulation.

Every module follows the same rhythm, so once you've done one, you'll always know what's coming next:

Step	What happens
Concept	One idea, explained clearly, with real examples
Why it matters	A real reason this shows up in actual data engineering work
Manual lab	You build something by hand — this is where the real learning happens
AI-assisted round	You extend it again with an AI assistant, and your job is to catch what it gets wrong
Common mistakes	The mistakes engineers actually make, explained in enough depth to recognize them again
Interview drill	Four rounds of questions — recall, judgment, debugging, AI-review

The nine modules, in order:

Project structure, packaging, dependency management
Typing and code quality as a safety net
Error handling, retries, resilience
Idempotency and safe reprocessing
Testing data pipelines, not just functions
Logging, observability, debuggability
Concurrency, async, working at scale
Configuration, secrets, environments
Packaging for deployment, CI/CD

Then: capstone integration, followed by a final interview simulation covering six rounds — system design, unfamiliar-code debugging, rapid-fire judgment, code review, AI-review, and behavioral.

Pace: roughly one module every 4-7 days for a working learner. Total course length: 8-10 weeks.

One project, growing the whole time. Your capstone is not a separate final assignment bolted on at the end. It's assigned on day one, and each module adds one real capability to the same running project. By Module 9, it's a complete, production-grade system — built by you, one deliberate piece at a time.

How to get the most out of this course

A few habits matter more than anything else for how much you'll actually get out of this course:

Always do the manual lab before the AI round. Every time, no exceptions. The manual pass is where your own judgment gets built. If you skip straight to the AI assistant, you'll have nothing to check its work against.
Don't skip the "make it break" step. Every module has a point where you deliberately break working code to see how it fails. This feels like extra work. It's actually where a lot of the real learning happens — production code is defined by how it fails, not just how it succeeds.
Keep your AI guardrail log honest — including when the AI got it right. The log isn't a list of failures. It's evidence of your review discipline. A module where you checked carefully and found nothing wrong is still worth a line.
Start your story-collection document today, not in week 8. Real behavioral stories fade from memory fast. Capture them while they're fresh — you can always polish the wording later.
Never let your capstone and your interview drills mix. Every drill uses scenarios and code you've never seen before, on purpose. If a drill ever feels like it's quietly reusing your own capstone code, that's a mistake in the drill, not a shortcut worth taking.
Treat the final deliverable as rehearsal, not a demo video. You're not making a highlight reel. You're practicing defending real decisions out loud, the exact skill an interview actually tests.

Your capstone project

Here is a scenario. Imagine this is your first week at a new job.

Your manager says:

Output / Note

"We have a third-party API. It gives us transaction records. We need those records, every day, safely stored in our Postgres warehouse. Can you build that pipeline?"

Sounds simple, right? Now here is the twist — the same twist you will face in almost every real data engineering job.

This API is not friendly. It is exactly like the messy, unreliable systems you will meet in the real world:

It paginates — you don't get everything at once, you get it in pages, one request at a time.
It rate-limits you — ask too fast, and it says "slow down."
It times out — sometimes, for no clear reason, it just doesn't respond.
It sends broken records — every so often, a record is missing a field, or has the wrong type of data.

This is not a made-up problem for the course. This exact API — flaky, unpredictable, annoying — will be running on your own laptop from Day 1 of Module 1 onward. You will spend the next nine modules learning how to build a pipeline that survives it.

So what exactly are you building?

In plain words: pull the data, check it, clean it, save it safely.

More precisely:

Ingest all records from the API, for a given date
Validate them — check that they are actually correct
Transform them — shape them the way your warehouse expects
Upsert them into Postgres (upsert means: insert if new, update if it already exists)

The one rule that matters more than any other

Read this part twice. It is the heart of the whole capstone.

Your pipeline must be safe to run more than once.

Ask yourself — why would you ever run the same pipeline twice?

Maybe it crashed halfway, and you had to restart it.
Maybe someone accidentally triggered it again.
Maybe you're just testing, and you ran it by mistake.

In all these cases, running it again — or restarting it midway — should never create duplicate rows. It should never corrupt your data.

This one idea is called idempotency, and you will hear this word a lot in this course. Almost every module you study — types, error handling, testing, logging — exists to help you protect this one guarantee. Keep this question in your mind as you go through the course:

Output / Note

"If I ran this twice, right now, would anything break?"

If the answer is ever "I'm not sure" — that's a signal. Something needs fixing.

Your checklist — what "done" looks like

You don't need to understand any of these terms yet. That is exactly what each module will teach you. Think of this table as a map. Whenever you feel lost, come back here and see where you are.

#	What you'll build	Which module teaches it
1	A proper, installable Python package with a command-line tool	Module 1
2	Fully typed code (`mypy --strict`), with pydantic checking your data at every boundary	Module 2
3	Errors sorted into clear categories, smart retries, and a safe place for records that fail	Module 3
4	Proof — an actual test — that running your pipeline twice changes nothing extra	Module 4
5	Real tests: unit tests, integration tests, and tests for when things go wrong	Module 5
6	Logs that actually help you — with a run ID, so you can trace what happened	Module 6
7	Fast, safe concurrency — fetching and processing many records at once, without breaking anything	Module 7
8	Configuration that is typed, checked, and never hardcodes a secret	Module 8
9	A CI pipeline — automatic checks that stop bad code from being merged	Module 9
—	An AI guardrail log — one line per module, growing as you go	Modules 1-9

By the time you finish Module 9, every single row above is done. Not "mostly done." Done. This is the exact bar that separates a hobby script from a production system — and it's the same bar an interviewer is quietly checking for when they ask, "walk me through a project you've built."

Learning to work with AI — the right way

Let's be honest for a moment.

Every one of you will use an AI coding assistant. Claude Code, Cursor, Copilot — it doesn't matter which one. That is not the question this course is asking.

The real question is:

Output / Note

Do you understand the code well enough to catch it when it's wrong?

This is the actual skill companies are hiring for now. Not "can you write code" and not "can you use AI." Both. Together.

Two ways of using AI — only one of them works

Way one: vibe coding. You type a prompt. AI gives you code. You paste it in. You run it. If it looks fine, you move on.

Sounds fast, right? It is fast. It is also how bugs quietly slip into production, sit there for weeks, and then blow up at the worst possible moment.

Way two: the closed loop. This is what we teach in this course, and it looks different:

You ask the AI to do something specific.
The AI proposes a plan, or writes code.
You run it. You look at the real output. Or the test fails, and you see exactly how.
You review the plan and the diff — like you're reviewing a teammate's pull request.
You iterate, based on what actually happened — not on how confident the AI sounded.

Notice the difference? In the closed loop, you never accept code just because it "looks right." You accept code because you watched it prove itself.

This course only practices the closed loop. Every single AI-assisted round, in every module, follows this pattern.

The guardrail log — your proof that you're doing this right

Starting from Module 1, every module gives you a real, known mistake that AI assistants tend to make on that exact topic. Not a vague "be careful." A specific, real pattern — the kind of mistake that has actually shown up when engineers tested these tools.

All of this gets written into one file: AI_GUARDRAIL_LOG.md, sitting at the root of your capstone repo, right next to SPEC.md. You'll create it during Module 1's capstone tie-in, and add to it every module after that — it's a running log, not something you rewrite each time.

Your job in every module:

Give the AI a task.
Require it to write and run tests before you accept anything.
Read the diff carefully — like reviewing a stranger's code.
Check: did it make the mistake we warned you about?
If yes — fix it, and write down one line: what went wrong, and the guardrail you're now using to catch it next time.

By the end of the course, you will have at least nine of these lines. Nine real moments where you caught a mistake a machine made. Do you know what this becomes? Direct proof, in an interview, that you can work with AI without switching off your own judgment. That is worth more than you might think right now.

What's already built for you, and what you'll build yourself

Already built. You don't need to rebuild it:

mock-api — the flaky third-party API we talked about earlier. Learn how it behaves by sending it requests, not by reading its code. That's exactly how you'd treat a real external API you don't control.
postgres — your warehouse database. The tables are already created for you.
Both come up with a single command: docker compose up -d. If that's not already working smoothly for you, stop here. Go back to SETUP.md and fix it first. Nothing else in this course works without this piece in place.

Yours to build, starting Module 1:

Everything else. The package. The types. The retries. The tests. The logs. The concurrency. The config. The CI pipeline. All of it grows, piece by piece, inside the same repo — starting from a small, simple skeleton in Module 1.

Start this today, not in the final week

Here is one more thing to begin right now, even before Module 1.

Start a simple document — just bullet points is fine — and begin collecting real stories from your own work. You need 3 to 4 of them. Think about:

A time something broke in production, and you had to fix it
A design decision you made that, later, turned out to be wrong
A disagreement you had with a teammate about how to build something
A time you made something faster or more efficient

Why start now, instead of waiting until the end of the course?

Because these stories are the raw material for your behavioral interview answers later. And memory fades. A story you remember clearly today will feel foggy in eight weeks. Capture it now, while it's fresh. You can always shape it into a polished answer later — that part is easy. The hard part is remembering the details, and that only gets harder with time.

What comes next

Two quick things before you move to Module 1:

If you haven't already, confirm your environment is working. Go through SETUP.md, then AI-ASSISTANT-SETUP.md, and make sure Docker, Postgres, the mock API, and an AI coding assistant are all running. This is not optional — everything else in this course depends on it.
Start your story-collection document. Even three bullet points is a good start.

Once both of these are done, you're ready. Head to Module 1, and let's start building.