Module 8: Configuration, Secrets, Environments

What you'll learn in this module

By the end of this module, you'll be able to:

Explain what configuration, secrets, and environment-specific settings actually are, and why they need different handling
Recognize the common, real ways teams manage these badly, and the specific damage each pattern causes
Use typed, validated settings that fail immediately when something required is missing — instead of quietly limping along
Explain, with real reasoning, why pydantic-settings is the right tool for this job — not just how to use it
Spot the single most common AI mistake with configuration: hardcoding a plausible-looking secret or default, even in a project that already has an established, correct pattern for this

Let's begin.

Let's start with a question

If someone handed you a project right now and asked "what does this need to actually run — what values, what credentials, what settings?" — could you answer confidently, just by looking at how the code is currently written?

For most real projects, the honest answer is no. The values a program needs to run are usually scattered — some in a file, some hardcoded deep in a function, some assumed to already exist as environment variables nobody documented. Nobody set out to make it this confusing. It happens gradually, one quick os.environ.get() call at a time, until the actual answer to "what does this need" lives nowhere you could point to.

This module is about fixing that — not just the specific danger of a leaked password, but the whole discipline of knowing, in one place, exactly what your program needs to run, and what happens when something's missing.

Why this matters in data engineering

Let's define three things clearly, because this module mixes them together constantly, and the differences actually matter.

Configuration is anything that changes how your program behaves, without changing your code. A timeout. A batch size. Which API to call. The same source code, run with different configuration, does meaningfully different things.

Secrets are a specific, sensitive kind of configuration — a database password, an API key, a signing token. Everything true about configuration is true about secrets too, plus one more thing: if a secret leaks, something bad happens to someone. That extra stakes level is why secrets deserve more care than configuration in general, not just the same care.

Environments are the different places the exact same code runs — local, staging, production — each one needing different configuration values for the same settings. Your database host in local development genuinely should be different from production. That's not inconsistency; it's the whole point of having environments at all.

Now ask yourself: in a real data pipeline, how many of these does your code actually touch? Which database to write to. Which API to call, and with what credentials. How many records to batch. Whether to run in debug mode. Almost every real pipeline depends on all three of these — which means getting configuration wrong doesn't fail loudly in one obvious spot. It fails wherever that specific piece of configuration happens to be used, often nowhere near where the actual mistake was made.

The core idea: how this normally goes wrong, and why each way is a real problem

Before getting to the right way to do this, it's worth naming the common wrong ways clearly — not as abstract mistakes, but as patterns with real, specific consequences.

Wrong way 1: hardcoding values directly in the code

The fastest way to get something running: just write the value straight into the source. db_host = "prod-db.internal". Works immediately. The problem shows up the moment you need a second environment — now the value has to change every time you deploy somewhere different, and someone has to remember to change it back, every single time, forever.

Wrong way 2: secrets committed to version control

A password typed directly into a config file that gets committed to git. This feels harmless in a private repo, right up until the repo's visibility changes, or a teammate forks it, or — this is the part people miss — the secret stays in git's history forever, even after someone "removes" it in a later commit. Deleting the line doesn't delete the leak.

Wrong way 3: configuration scattered across the codebase

os.environ.get("SOME_VALUE") called in twelve different files, each with its own default, or no default, decided independently by whoever happened to write that file. Nobody can answer "what does this project need to run" by reading one place — they'd have to read the entire codebase and mentally collect every scattered call. This is exactly how a whole required value can go quietly missing without anyone noticing until it actually breaks something.

Wrong way 4 — the one this module spends the most time on: a fallback value for something that should be required

This is worse than the other three, and worth understanding precisely why. Picture this pattern, which looks completely reasonable at a glance:


python
password = os.environ.get("DB_PASSWORD", "dev_only_pw_2019")

This doesn't crash. It doesn't complain. If the real password is missing — a deploy misconfiguration, a secret that failed to mount — the program just quietly uses the fallback and keeps running. Ask yourself: what's actually worse, a program that refuses to start when misconfigured, or one that starts anyway and does something silently wrong? The crash is annoying but cheap — someone gets paged in seconds, the fix is usually obvious. The silent wrong version can run for hours, producing plausible-looking, completely incorrect results, with nothing anywhere telling you it's wrong. That gap — between a loud, cheap failure and a quiet, expensive one — is the single most important idea in this whole module.

So what does "doing this right" actually require?

Look back at those four failure patterns, and notice they share a root cause: nothing forces configuration to be declared, typed, and checked in one place. Fixing this for real means all of the following, together, not just one of them:

One place that lists everything the program actually needs — not scattered across a dozen files
Real types — a port number should be an int, not whatever string happened to be sitting in the environment
A hard line between what's genuinely safe to default and what never is — a timeout can default to 30; a production database password cannot default to anything
Failure at startup, not failure hours later — if something required is missing, the program should refuse to run, immediately, with a clear message saying exactly what's missing

Why `pydantic-settings` is the right tool for this, specifically

This is where pydantic-settings comes in — and it's worth understanding why it fits, not just how to use it. It gives you exactly the four things above, together, as one mechanism instead of four separate disciplines you have to remember to apply consistently:

One class describes every setting your program needs — a single, readable answer to "what does this need to run."
Real type validation, the same way pydantic validates a data model — a setting declared as int that receives "not-a-number" fails immediately, with a clear error, not a confusing crash three functions later.
A field with no default is required, and pydantic enforces that automatically — you don't have to remember to add a check; leaving off a default is the check.
It fails at the moment the settings object is created — right at startup, before any real work happens, which is exactly the "loud and cheap" failure this module has been building toward, instead of the "quiet and expensive" one.

The manual lab below shows this precisely: the same hardcoded-fallback mistake from "wrong way 4" above, and the fix, expressed as one missing default value.

Manual lab: the report that was wrong for hours

Getting the lab files

Download module-8-materials.zip:


bash
cd ~/courses/pp4de
cp /mnt/c/Users/yourname/Downloads/module-8-materials.zip course-materials/
cd course-materials
unzip module-8-materials.zip

Commit it:


bash
cd ~/courses/pp4de
git add course-materials/module-8
git commit -m "Add Module 8 lab materials"
git push

The scenario

You've inherited a small internal tool that generates a quarterly revenue report from a database. It's been running fine for two years. Then, during an infrastructure migration, the real database password briefly stops getting injected into one environment for a few hours — and nobody notices, because the tool never actually stops working.

Reproduce the problem first

Go into the starter folder:


bash
cd course-materials/module-8/starter

Run it, without setting anything:
```
bash
python3 run.py
```
Read the output. It looks completely normal — a clean connection message, a nicely formatted quarterly revenue report, no errors anywhere.
Now open report_db.py and look at _REAL_REVENUE_TOTALS versus _WRONG_ENVIRONMENT_TOTALS. Compare those numbers to what you just saw printed. You were looking at the wrong dataset the entire time — and nothing about the output told you so.
Open config.py. Find get_db_password. There's your answer: a hardcoded fallback, "dev_only_pw_2019", silently used whenever the real environment variable isn't set. This is exactly "wrong way 4" from this module's concept section, not a hypothetical.

Sit with what just happened. The script ran cleanly. It produced a plausible, correctly formatted report. And every number in it was wrong, with nothing anywhere telling you that.

Your task

Fix this so that:

The database password has no fallback at all — if it's not provided, the service should refuse to start.
Missing the password produces a clear, immediate error naming exactly what's missing — not a confusing failure somewhere else, and not silent wrong behavior.
Everything still works correctly when the real password is actually provided.
Config is defined in one typed place, not scattered os.environ.get(...) calls.

A question worth asking before you look at the solution

Same habit as every module. If the solution looks different, ask why.

Here's the honest answer: the actual reporting logic — connect, fetch revenue, print it — doesn't change at all. What changes is entirely about the shape of the failure when something required is missing: from silent and wrong, to loud and immediate. That's the whole fix.

Full worked solution

The complete solution lives in course-materials/module-8/solution/:

solution/
├── pyproject.toml
├── run.py
└── src/
    └── reportsvc/
        ├── __init__.py
        ├── report_db.py    # unchanged - the simulated database
        └── config.py       # the fix: typed settings, no fallback for the secret

config.py — the actual fix:


python
from pydantic_settings import BaseSettings, SettingsConfigDict


class ReportServiceSettings(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="REPORT_DB_")

    host: str = "localhost"
    password: str

Notice the difference between these two fields. host: str = "localhost" has a default — genuinely reasonable, since localhost is a sensible guess for local development, and getting it wrong just means a connection failure, not silently wrong data. password: str has no default at all. That absence is the entire fix. Pydantic treats a field with no default as required — creating a ReportServiceSettings without a REPORT_DB_PASSWORD environment variable set raises an error immediately.

run.py — barely different from the starter, on purpose:


python
from reportsvc.config import ReportServiceSettings
from reportsvc.report_db import fetch_quarterly_revenue


def main() -> None:
    settings = ReportServiceSettings()

    print(f"Connecting to {settings.host}...")
    revenue = fetch_quarterly_revenue(settings.host, settings.password)

    print("Quarterly revenue report:")
    for quarter, total in revenue.items():
        print(f"  {quarter}: ${total:,.2f}")


if __name__ == "__main__":
    main()

A real gotcha worth knowing about: `mypy --strict` and settings classes

Here's something you'll likely hit yourself, worth knowing in advance. Run mypy --strict against a naive version of this fix, and you'll probably see something like:

error: Missing named argument "password" for "ReportServiceSettings"

This looks like a real problem — ReportServiceSettings() is called with no arguments anywhere in run.py. But it isn't one: pydantic-settings populates fields from environment variables at runtime, not from constructor arguments — mypy, by default, doesn't know that, and treats password as a normal required constructor argument that was never supplied.

The real fix isn't to silence this with a # type: ignore. It's telling mypy the truth about how this library actually works, using pydantic's own mypy plugin:


toml
[tool.mypy]
strict = true
plugins = ["pydantic.mypy"]

With the plugin enabled, mypy --strict passes cleanly — it now understands that BaseSettings fields are populated differently than a normal class's constructor arguments.

Verify it, step by step

Create a virtual environment and install:


bash
cd course-materials/module-8/solution
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install mypy

Run mypy --strict:


bash
python -m mypy --strict src/reportsvc/ run.py

Expected: Success: no issues found.

Run it without setting the password, and confirm it fails immediately, loudly, with a clear message:
```
bash
python run.py
```
You should see a pydantic_core.ValidationError mentioning password and Field required — not a report, not a silent wrong answer.
Now run it with the real password set:
```
bash
REPORT_DB_PASSWORD="vault-injected-prod-secret-9f3a" python run.py
```
You should see the real revenue numbers — Q1: $482,910.00 and so on — clearly different from what you saw the starter code print in step 2 of "Reproduce the problem first."

If step 3 failed loudly instead of running quietly wrong, and step 4 produced the genuinely correct numbers, you've verified the actual thing this module set out to teach.

AI-assisted round

The task

Ask your assistant to extend the config with a new setting:

Output / Note

"Add a setting for the report service's API key, used to send the generated report to a third-party notification service. Wire it into ReportServiceSettings, and write a test confirming the service fails clearly if it's missing. Run the test before telling me you're done."

The known failure pattern to watch for

This is one of the most common AI mistakes with configuration, named directly in this module's own concept section: hardcoding a plausible-looking secret or default, even in a project that already has an established, correct pattern to follow.

Here's what that tends to look like:


diff
  class ReportServiceSettings(BaseSettings):
      model_config = SettingsConfigDict(env_prefix="REPORT_DB_")

      host: str = "localhost"
      password: str
+     api_key: str = "test-api-key-12345"

Notice this sits in the exact same class as a field that's already correctly required, with no default — password. The assistant had a working pattern right there to copy, and instead reached for a plausible-looking placeholder default for the new field, quietly reintroducing the exact bug this whole module exists to fix, one field away from the fix itself.

The guardrail

If you catch this, write it down:

Output / Note

Module 8 guardrail: Any time an AI assistant adds a new configuration field, check specifically whether it gave it a default — and if that field represents a secret or a required connection target, a default of any kind is the wrong choice, even a plausible-looking one. Check this against fields that are already correct in the same class; a new field with a default sitting right next to a required field with none is a strong signal something was missed.

If your assistant correctly left the new field required, matching the existing pattern, without being told to — log that too.

Common mistakes

Any fallback value for a required secret, no matter how "dev-only" it looks. The presence of a plausible-looking fake credential in source code is itself a secrets-hygiene problem, independent of whether it ever actually gets used.
Treating "the service didn't crash" as evidence it's configured correctly. A quiet, wrong service is not a working service — it's a more dangerous kind of broken.
Scattering os.environ.get(...) calls throughout a codebase instead of centralizing config in one typed place — makes it easy to lose track of what's actually required versus genuinely optional.
Silencing a real mypy error about settings classes with a blind # type: ignore, instead of understanding why it's happening and fixing it properly with the right plugin or configuration.

Capstone tie-in

Step 1: Open your capstone repo and start the infrastructure

Confirm the title bar says pp4de [WSL: Ubuntu].


bash
docker compose start
docker compose ps

Step 2: Build typed settings for your capstone

In VS Code's Explorer:

Right-click pipeline (inside src/) → New File → name it settings.py.
Build a BaseSettings class covering your capstone's real configuration — the mock API's base URL (safe to default, since it's not a secret), and your Postgres connection details (host, port, database name can default; the password should not).
Add pydantic-settings to pyproject.toml's dependencies, and add the pydantic.mypy plugin to your [tool.mypy] section, the same way as this module's lab.

Step 3: Prove the fail-fast behavior against your real Postgres password

Open the integrated terminal, confirm your venv is active, and deliberately unset your database password to prove the fail-fast behavior works on your actual capstone, not just the lab:


bash
env -u POSTGRES_PASSWORD python3 -c "from pipeline.settings import Settings; Settings()"

You should see a clear ValidationError naming the missing field — not a silent fallback, and not a confusing unrelated error somewhere else.

Step 4: Commit and push through VS Code's Source Control panel

Click the Source Control icon in the sidebar.
Confirm .venv and .env are not listed under Changes — .env holds your real local secrets and should never be committed, the same rule from setup.
Stage, commit with a message like Module 8: typed, fail-fast configuration, and click Commit.
Click Sync Changes to push to GitHub.
Confirm the changes appear on GitHub.

Check row 8 off your SPEC.md checklist. One to go.

Before you close for the day:


bash
docker compose stop

Interview drill

Every question below follows the same pattern. First, the question. Then, what the interviewer wants to hear. Then, a junior engineer's answer. Then, a strong senior answer.

Recall

Question: "Why is a hardcoded fallback for a required secret actually worse than the service crashing on startup?"

What the interviewer wants to hear: a real comparison of the two failure modes — not just "hardcoding secrets is bad," but specifically why the fallback behavior is more dangerous than a crash.

Junior answer: "Hardcoding secrets is a bad practice — you shouldn't put passwords directly in code."

(True, but doesn't actually answer the question — doesn't compare the fallback's failure mode to a crash, which is what's actually being asked.)

Senior answer: "A crash on startup is loud, immediate, and cheap — a deploy fails, someone gets paged in seconds, and the error message usually says exactly what's wrong. A hardcoded fallback turns a missing required secret into a silent failure instead — the service keeps running, using a value that's either wrong or points somewhere unintended, and nothing about 'the service is up' signals that anything's wrong. That's strictly worse, because the cost of the bug scales with how long it goes unnoticed — hours or days of real work happening against the wrong thing, instead of a failed deploy that gets fixed in minutes. The fix isn't just 'don't hardcode this specific value' — it's that required configuration should have no fallback at all, so a misconfiguration fails as early and as loudly as possible."

Debugging

Question: shown as a real snippet and incident story, with no explanation yet:


python
import os

def get_db_connection():
    host = os.environ.get("PAYROLL_DB_HOST", "localhost")
    password = os.environ.get("PAYROLL_DB_PASSWORD", "dev_only_pw_2019")
    return connect(host=host, password=password, user="payroll_svc")

"This code has been unchanged for two years. Last month, during a Kubernetes namespace migration, a secret mounting misconfiguration meant PAYROLL_DB_PASSWORD wasn't actually injected into one specific pod for about 6 hours. The service didn't crash. It didn't even error. It just... quietly kept running. Why is that worse than a crash, and what does it reveal about this code?"

What the interviewer wants to hear: identifying that the real problem is the presence of a fallback for a required secret at all — not just "the password is hardcoded" — and connecting it to typed, validated config as the structural fix.

Junior answer: "The password shouldn't be hardcoded in the source code — it should come from an environment variable instead."

(Misses that the code already does read from an environment variable — the actual problem is the fallback that activates when that variable is missing, not the mechanism of reading it.)

Senior answer: "The real problem isn't that the password is hardcoded in isolation — it's that a required piece of configuration has a fallback at all. Some config genuinely has safe defaults, like a port number or a timeout. Credentials and connection targets for production data generally don't. When PAYROLL_DB_PASSWORD wasn't injected, this code didn't fail — it silently substituted a plausible-looking but wrong value and kept running, for six hours, before anyone noticed. A crash on startup would have caught this in seconds. The fix is failing fast: no default for required secrets, raise immediately if it's missing. I'd also connect this to typed, validated config — a pydantic settings model that makes this field required with no default — rather than ad hoc os.environ.get calls scattered through the codebase, since that structurally prevents this exact mistake from being possible, instead of relying on someone remembering not to add a fallback next time. Worth noting too: even the mere presence of a string like dev_only_pw_2019 in application code is its own secrets-hygiene smell, independent of whether this specific incident had happened."

AI-review

Question: shown this diff, with no explanation yet:


diff
  class ReportServiceSettings(BaseSettings):
      model_config = SettingsConfigDict(env_prefix="REPORT_DB_")

      host: str = "localhost"
      password: str
+     api_key: str = "test-api-key-12345"

What the interviewer wants to hear: recognizing that the new field reintroduces the exact fallback-for-a-secret problem this module is about, especially notable because it sits right next to a field that's already correctly required.

Junior answer: "The naming is a little inconsistent — api_key probably shouldn't have a hardcoded test value in it, but it's a minor style thing."

(Correctly notices something's off but treats it as a minor style issue rather than the same category of real risk as the password field right above it.)

Senior answer: "This is the exact same problem this whole module is about, reintroduced one field below the fix. password has no default — correctly required, fails fast if missing. api_key has a default, 'test-api-key-12345' — a plausible-looking placeholder that will silently activate if the real API key isn't set, exactly the fallback behavior we just eliminated for password. If this API key is used to send reports to a real third-party service, a missing real key means requests either fail with a confusing auth error somewhere downstream, or worse, actually succeed against some test tier the placeholder happens to be valid for — either way, silently, not loudly. I'd tell the assistant to remove the default entirely, matching the password field right above it — there's no reason this class should treat two secrets differently, especially when the correct pattern is sitting right there to copy."

Judgment

Interviewer: "You're an engineer on a payroll platform, working on a service that processes employee timesheets. Your tech lead proposes adding a fallback default for the database connection string, so that if the config service is ever briefly unavailable during deployment, the service can still start up using a cached last-known-good value, instead of failing to deploy. What do you say?"

Candidate (asking first): "Before I answer — what would that cached value actually point to? Is it always guaranteed to be the correct, current production database, or could it, even rarely, point somewhere stale or wrong?"

Interviewer: "Good question — it would be whatever value was last successfully fetched, cached locally. In the normal case, that's correct. But if there'd been a recent, legitimate database migration or credential rotation between deployments, that cached value could be stale."

Candidate (first answer): "Then I'd push back on this, even though I understand the deployment-reliability motivation. This is exactly the pattern this platform should be most careful about — a payroll service silently connecting to a stale or wrong database is a much worse outcome than a deploy failing loudly. A failed deploy is annoying and gets fixed in minutes. A payroll service quietly writing to the wrong database, or reading stale credentials, could mean real financial data gets processed incorrectly, for real employees, before anyone notices."

Interviewer (pushback): "But deployment reliability matters too — isn't a service that can't start at all also a real business problem, maybe an even more visible one?"

Candidate (round 1): "It is a real problem, but I'd separate two different failure modes here, because they're not equally bad. Deployment reliability being interrupted is loud, visible, and safe — everyone knows immediately that something's wrong, and no incorrect work happens while it's being fixed. A stale-fallback connection string is quiet and potentially unsafe — the service looks fine, appears to be running normally, and might be doing real, incorrect work against payroll data the whole time. I'd rather have a loud, safe failure than a quiet, unsafe success, especially on a system handling people's pay."

Interviewer (second pushback): "So how would you actually solve the underlying reliability concern, if not with a fallback?"

Candidate (round 2): "I'd address deployment reliability directly, rather than through a fallback on the connection string itself. That could mean making the config service itself more reliable — redundancy, retries with real backoff on the deploy tooling's side, not the application's — or building in a deliberate, short retry window during startup specifically for transient config-service unavailability, that still fails loudly and clearly if it can't get a fresh value within that window. Either of those improves reliability without ever letting the service silently run on a value that might be stale or wrong."

Interviewer (final challenge): "Your tech lead says a short retry window sounds basically the same as what they proposed — convince them it's actually different."

Candidate (final defense): "The key difference is what happens when it ultimately fails. A retry-with-timeout-then-fail approach still ends in a loud, safe failure if it can't get a fresh, correct value — the service simply doesn't start, and that's the correct outcome if we genuinely can't confirm we have the right configuration. A cached fallback approach ends in the service starting anyway, on a value we can't fully vouch for. Both approaches try to smooth over a brief config-service hiccup, but only one of them guarantees we never silently run payroll processing against configuration we're not actually sure is current and correct. That distinction — fail loudly versus succeed uncertainly — is the whole reason I'd insist on the retry-then-fail version instead."

Model strong answer — a single answer, given upfront:

Output / Note

"I'd want to know first whether that cached fallback value is always guaranteed correct, or whether it could ever be stale — for example, after a database migration or credential rotation between deployments. Assuming it could be stale, I'd push back on this proposal, even though I understand the deployment-reliability motivation behind it. On a payroll platform specifically, a service that silently starts up using a possibly-stale database connection is a much worse outcome than a deploy failing loudly — a failed deploy gets noticed and fixed in minutes, while a quiet connection to the wrong database could mean real financial data gets processed incorrectly for real employees before anyone realizes. I'd address the actual reliability concern directly instead — either making the config service itself more resilient, or adding a short, bounded retry window during startup for transient unavailability, one that still fails loudly and clearly if it can't confirm a fresh, correct value within that window. That improves deployment reliability without ever letting the service silently run on configuration we can't fully vouch for — the goal isn't zero failures, it's failures that are loud and safe instead of quiet and uncertain."